Hello all,
I'm not sure if any of you will of even heard of this ISP. They are called Lumison and have datacenters in both Milton Keynes and Edinburgh. We are an ITC consultancy firm and support many businesses throughout the UK and therefore manage and advise many companies ITC and data communications. We deal with many ISP's and regularly keep up with ISPreview's news and stats. Anyway, back to Lumison...
You can see from their website they claim to be one of the UKs largest ISPs but we're struggling to find any information that supports this claim. They certainly are not in the top 10 from what we can see. That's just the first and most minor point. if anyone can shed any light we'd be interested to hear from you.
http://www.lumison.net/why-lumison
Established in 1995, Lumison has grown from a local dial up provider into one of the UK’s largest, most trusted full-service ISPs with literally thousands of SME, Corporate and Government clients.
In terms of their service we have found them to be by far the poorest in terms of reliability and support.
While they claim to provide a 24/7 service in terms of connectivity and support we have recently seen them 'fall off the map, and lost ALL forms of communication after power testing in their datacenter.
http://www.lumison.net/products-solutions/hosting/colocation-and-datacentres
The outage:
Whilst testing a power circuit in business hours at their datacenter in Edinburgh a switch failed to operate once the power came back on. The affect of this caused:
The loss of all Leased Lines and broadband services to all their customers.
Their own 'single hosted' website in their datacenter was unavaliable too so no status reports.
All their voice comms were down because again all hosted in their datacenter.
All backup lines were down.
ALL VPN's down.
All voice comms provided to their business clients down.
All forms of communication to their support teams down.
All email services down.
All websites and hosted services down.
All mobile and SMS communication down so again no service status reports or notifications.
all for over 4 HOURS during business hours!
As you can imagine this caused absolute havoc for our clients that rely on their services. Many of our clients suffered significant financial losses due to their inability to trade and communicate costing many of them many thousands of pounds in lost revenue. What made matters worse was that no one was able to contact Lumison to find out what the situation was. This caused even more problems for supporting IT and Comms engineers. Businesses even started checking to see if Lumison had possibly gone into liquidation it was that bad. They fell off the map!
2 weeks later our clients have started to recieve credit notes for the loss of service. One of which kindly informed us of the actual credit amount. Bear in mind that this client pays Lumison in excess of £1000.00 per month for Leased Line and broadband services.
Credit note from Lumison:
Credit for the outage on Wednesday 28th September, covering a period of four hours of
downtime. This has been calculated on an hourly basis per service as per the terms of the Service Level
Agreement.
Total excl VAT: £4.35
You might imagine how our clients are reacting to this!
Lumison's feedback on the outage:
In response to feedback and queries from our customers, we would like to update you on the recent network outage issue at our Newbridge datacentre, the process of applying credit notes to those affected, and the measures we are taking to improve our internal processes on DR and communications.
Credit Notes
Contracted credit notes will be applied to customer accounts automatically over the coming week, in line with your SLAs across effected services. In recognition of the inconvenience caused, we have taken the longest affected service period and rounded it up to 4 hours. The majority of customers were affected for a shorter period of time but we feel this is the fairest approach.
Scheduling of the Testing
We are under regulatory obligation by Scottish Power to perform systems testing during Monday to Friday and are restricted in the timing of that testing. Since the datacentre was built over 6 years ago, these biannual tests have occurred at similar times without incident.
Telephony System
Our phone system is hosted within our Newbridge datacentre and has failovers and multiple network links to a PBX gateway. There is also a backup system which was tested within the last month. Redundancy exists through the system to deal with component and software failures but in this instance, no network traffic was getting out of Newbridge due to the issue.
If we encounter issues like those we experienced on Wednesday, our DR plan states that our telephony system is manually re-directed to our Milton Keynes datacentre to enable our main communication link to remain operational. Initially, our processes for this manual re-direction did not work and we took longer than expected to get a successful number redirection.
The Milton Keynes support line operates at a lower maximum capacity than the Newbridge site, and this affected the ability of some of our customers to access telephone support.
The level of resilience already within the telephony system is under review to expand the level of protection with automated failover.
SMS
As a business, we understand that a widespread outage could interrupt our customers’ ability to reach websites, receive email, and may affect the ability to receive telephone calls. As a result, we operate a text alerting system. This system operates via an internet-based gateway to mobile providers with a secondary gateway that does not use our network, but connects via mobile providers’ GSM network.
Despite undergoing recent testing, the backup did not function as expected during the incident which meant that our communication did not go out as normal. To address this issue, we will be reviewing the out-of-band solution and are planning to upgrade to a more advanced system with greater resiliency in the next few weeks.
Service Updates
For customer service and internal performance reasons, the Lumison website is hosted in our network and is within our Newbridge datacentre. As a result, service updates were unavailable to customers. Plans are in place to split out the service update section of our website and make it available via a secondary, separately-hosted service update facility.
Updates should have been provided through our main Twitter account and we are currently reviewing our DR communications policy to cover the use of social media channels.
We would like to apologise again to customers for the outage, which dropped our standard 100% core network service availability level to 99.95% over the last twelve months. Lumison BlueSquare remains committed to maintaining continuity to ensure our customers have the level and quality of service expected from the business.
Thank you.
Kind Regards.
Aydin Kurt-Elli
Group Chief Operating Officer
So as you can see a total failure of every part of their redundant systems and 100% failure of the Disaster Recovery procedures. FYI their SLA for their Leased Line services provides little to nothing to any of their customers should they fail to provide the service they are charging them for.
Our advice.
Quite obviously avoid Lumison, assuming you've ever heard of them.
Read and understand any SLA that comes with a service.
Assume your ISP does not have adequate Disaster Recovery procedures because what they profess and what is reality could be two entirely different things.
FYI: We have never recommended this ISP. Thankfully ^^
I'm not sure if any of you will of even heard of this ISP. They are called Lumison and have datacenters in both Milton Keynes and Edinburgh. We are an ITC consultancy firm and support many businesses throughout the UK and therefore manage and advise many companies ITC and data communications. We deal with many ISP's and regularly keep up with ISPreview's news and stats. Anyway, back to Lumison...
You can see from their website they claim to be one of the UKs largest ISPs but we're struggling to find any information that supports this claim. They certainly are not in the top 10 from what we can see. That's just the first and most minor point. if anyone can shed any light we'd be interested to hear from you.
http://www.lumison.net/why-lumison
Established in 1995, Lumison has grown from a local dial up provider into one of the UK’s largest, most trusted full-service ISPs with literally thousands of SME, Corporate and Government clients.
In terms of their service we have found them to be by far the poorest in terms of reliability and support.
While they claim to provide a 24/7 service in terms of connectivity and support we have recently seen them 'fall off the map, and lost ALL forms of communication after power testing in their datacenter.
http://www.lumison.net/products-solutions/hosting/colocation-and-datacentres
The outage:
Whilst testing a power circuit in business hours at their datacenter in Edinburgh a switch failed to operate once the power came back on. The affect of this caused:
The loss of all Leased Lines and broadband services to all their customers.
Their own 'single hosted' website in their datacenter was unavaliable too so no status reports.
All their voice comms were down because again all hosted in their datacenter.
All backup lines were down.
ALL VPN's down.
All voice comms provided to their business clients down.
All forms of communication to their support teams down.
All email services down.
All websites and hosted services down.
All mobile and SMS communication down so again no service status reports or notifications.
all for over 4 HOURS during business hours!
As you can imagine this caused absolute havoc for our clients that rely on their services. Many of our clients suffered significant financial losses due to their inability to trade and communicate costing many of them many thousands of pounds in lost revenue. What made matters worse was that no one was able to contact Lumison to find out what the situation was. This caused even more problems for supporting IT and Comms engineers. Businesses even started checking to see if Lumison had possibly gone into liquidation it was that bad. They fell off the map!
2 weeks later our clients have started to recieve credit notes for the loss of service. One of which kindly informed us of the actual credit amount. Bear in mind that this client pays Lumison in excess of £1000.00 per month for Leased Line and broadband services.
Credit note from Lumison:
Credit for the outage on Wednesday 28th September, covering a period of four hours of
downtime. This has been calculated on an hourly basis per service as per the terms of the Service Level
Agreement.
Total excl VAT: £4.35
You might imagine how our clients are reacting to this!
Lumison's feedback on the outage:
In response to feedback and queries from our customers, we would like to update you on the recent network outage issue at our Newbridge datacentre, the process of applying credit notes to those affected, and the measures we are taking to improve our internal processes on DR and communications.
Credit Notes
Contracted credit notes will be applied to customer accounts automatically over the coming week, in line with your SLAs across effected services. In recognition of the inconvenience caused, we have taken the longest affected service period and rounded it up to 4 hours. The majority of customers were affected for a shorter period of time but we feel this is the fairest approach.
Scheduling of the Testing
We are under regulatory obligation by Scottish Power to perform systems testing during Monday to Friday and are restricted in the timing of that testing. Since the datacentre was built over 6 years ago, these biannual tests have occurred at similar times without incident.
Telephony System
Our phone system is hosted within our Newbridge datacentre and has failovers and multiple network links to a PBX gateway. There is also a backup system which was tested within the last month. Redundancy exists through the system to deal with component and software failures but in this instance, no network traffic was getting out of Newbridge due to the issue.
If we encounter issues like those we experienced on Wednesday, our DR plan states that our telephony system is manually re-directed to our Milton Keynes datacentre to enable our main communication link to remain operational. Initially, our processes for this manual re-direction did not work and we took longer than expected to get a successful number redirection.
The Milton Keynes support line operates at a lower maximum capacity than the Newbridge site, and this affected the ability of some of our customers to access telephone support.
The level of resilience already within the telephony system is under review to expand the level of protection with automated failover.
SMS
As a business, we understand that a widespread outage could interrupt our customers’ ability to reach websites, receive email, and may affect the ability to receive telephone calls. As a result, we operate a text alerting system. This system operates via an internet-based gateway to mobile providers with a secondary gateway that does not use our network, but connects via mobile providers’ GSM network.
Despite undergoing recent testing, the backup did not function as expected during the incident which meant that our communication did not go out as normal. To address this issue, we will be reviewing the out-of-band solution and are planning to upgrade to a more advanced system with greater resiliency in the next few weeks.
Service Updates
For customer service and internal performance reasons, the Lumison website is hosted in our network and is within our Newbridge datacentre. As a result, service updates were unavailable to customers. Plans are in place to split out the service update section of our website and make it available via a secondary, separately-hosted service update facility.
Updates should have been provided through our main Twitter account and we are currently reviewing our DR communications policy to cover the use of social media channels.
We would like to apologise again to customers for the outage, which dropped our standard 100% core network service availability level to 99.95% over the last twelve months. Lumison BlueSquare remains committed to maintaining continuity to ensure our customers have the level and quality of service expected from the business.
Thank you.
Kind Regards.
Aydin Kurt-Elli
Group Chief Operating Officer
So as you can see a total failure of every part of their redundant systems and 100% failure of the Disaster Recovery procedures. FYI their SLA for their Leased Line services provides little to nothing to any of their customers should they fail to provide the service they are charging them for.
Our advice.
Quite obviously avoid Lumison, assuming you've ever heard of them.
Read and understand any SLA that comes with a service.
Assume your ISP does not have adequate Disaster Recovery procedures because what they profess and what is reality could be two entirely different things.
FYI: We have never recommended this ISP. Thankfully ^^























