UPDATE2 2nd Major Power Fault Affects UK ISPs, BT and Internet Services
After yesterday’s major disruption (here) we’re loathed to report that this morning appears to have started off in much the same way, with broadband ISPs from across the UK all reporting similar Internet disruption as a result of another possible power failure at a major London data centre.
Several ISPs are already pointing the finger at a power failure stemming from Telehouse North (specifically TFM10 as all the other kit is still powered). Meanwhile another ISP said they’ve lost all their London resilient interconnects to the BTWholesale DSL platform, although this has failed over to Manchester and so the impact was minimal.
The problem itself began after 7am this morning and once again it is affecting routing / DNS connectivity for multiple ISPs (i.e. you might have trouble accessing some websites and Internet services), albeit mostly those that use BT’s platform as a lot of ISPs connect through them. At the time of writing Telehouse are still working to restore power.
We should point out that yesterday’s fault stemmed from TeleCity, not Telehouse (i.e. different site), although it’s easy to get the two confused. No doubt questions will be asked about why this has happened twice in a row, albeit at different sites. Here’s a summary of some ISP updates.
https://twitter.com/fastbroadband/status/756036118215262208
UPDATE – The current outage affecting BT based connections is linked to a power failure at Telehouse North. There is no ETA yet, sorry
— ICUK.net (@icuknet) July 21, 2016
Sorry if you're unable to access some websites this morning on Plusnet Broadband, we're investigating the cause and will update you all soon
— Plusnet Help (@plusnethelp) July 21, 2016
The outage at BT may also be affecting some VOIP trunks that we supply; current ETA for this to be fixed is 10am.
— Merula Limited (@MerulaSupport) July 21, 2016
more internet woes. Yesterday power fail in Telecity. This morning part of Telehouse Nth out. Our kit currently unaffected.
— HiWiFi (@hi_wifi) July 21, 2016
A BT outage is affecting all ISPs. We are working with BT to ensure a fix is put in place ASAP!
— Zen Internet (@zeninternet) July 21, 2016
UPDATE 10:06am
We’re getting tentative reports that the situation is now being resolved, although like yesterday it could take a couple of hours before everything returns to normal. In some cases you might be able to get rid of the problem by power cycling (rebooting) your router, but don’t do this repeatedly as it may harm your service speeds.
UPDATE 12:29pm
Telehouse Europe has finally issued a brief statement on the matter: “We are aware that there has been an issue with the tripping of a circuit breaker within Telehouse North that has affected a specific and limited group of customers within the building. The problem has been investigated and the solution identified. Our engineers are working with our customers on the resolution right now. We will release updates in due course.”
Some equipment will need to be replaced, although most ISPs have already managed to move traffic around the problem area.
Mark is a professional technology writer, IT consultant and computer engineer from Dorset (England), he also founded ISPreview in 1999 and enjoys analysing the latest telecoms and broadband developments. Find me on Twitter, Google+, Facebook and Linkedin.
« UPDATE UK ISP TalkTalk Ranked Worst for Call Centre Customer Service
Comments are closed.
Privacy Notice: Please note that news comments are anonymous, which means that we do NOT require you to enter any real personal details to post a message. By clicking to submit a post you agree to storing your comment content, display name, IP, email and / or website details in our database, for as long as the post remains live.
Only the submitted name and comment will be displayed in public, while the rest will be kept private (we will never share this outside of ISPreview, regardless of whether the data is real or fake). This comment system uses submitted IP, email and website address data to spot abuse and spammers. All data is transferred via an encrypted (https secure) session.
NOTE 1: Sometimes your comment might not appear immediately due to site cache (this is cleared every few hours) or it may be caught by automated moderation / anti-spam.
NOTE 2: Comments that break our rules, spam, troll or post via known fake IP/proxy servers may be blocked or removed.
It’s the wrong kind of summer! 🙂
Wrong kind of wind, wrong kind of rain, wrong kind of summer.. when is it the right kind of anything anymore? 🙂
Perhaps the UK has the wrong kind of people?
4th – 6th August, Worcester Beer Fesitival…
… most DEFINITELY the right kind of beer!! 🙂
..unless it’s watered down 😉 .
I will aim to undertake some rigorous testing then and let you know 🙂
probably because the network needs a new revamp but they rather bodge it instead of fixing it but say no more its British Telecom and the cowboys
Does this, or yesterday’s issue, affect Virgin Media?
No, it doesn’t, as far I know. I feel sorry for those who have to use BT’s network.
Intrigued by Zen’s comment: “A BT outage is affecting all ISPs.”
It didn’t seem to impact EE 4G yesterday nor today. All working perfectly here.
Assuming that VM’s network is routed differently, it may not affect them either.
So it’s not “all ISPs”, is it.
But it does go to show that when it actually comes down to it and regardless of how much OFCOM might crow about success creating a competitive market, parts of the routing are the same, the BTW backhaul is the same for most, the cables to the property are the same, the engineers are the same..
There isn’t really much in the way of real competition at all as they share the same bottlenecks and failure points and so the customer has remarkably little choice.
Since when did a mobile operator become an ISP?
Why is it not an ISP?
We’ve used MBB for our office and home connection for about 6 years now. It’s the fastest and seemingly most resilient connection available here.
Fixed line only has relatively slow VDSL to offer, and there’s no cable.
@ Curious
Since the invention of the smart phone
This is a timely reminder of the lack of resilience by routing virtually all the UK’s internet traffic via one central hub. ISPs should be establishing alternatives in case of a major failure in London Docklands.
The lack of resilience is more individual companies investment rather than not being available. We interconnect with BT (and other suppliers) in both London and Manchester datacentres so all our customers automatically moved over to Manchester within 10 minutes of the failure this morning. The cost of having that capability and bandwidth is very high however and so a lot of ISPs fail to put in that level of investment due to costs being driven down by customers. You are right that a single datacentre is always a point of failure, no matter, how rare and so using multiple datacentres in multiple cities is paramount. Something any customers should consider when choosing a supplier.
That resilience is important, and expensive. Like anything that requires a subscriber to pay more per month, it isn’t a feature that sits high on people’s priority list until just after they’ve been hit by it.
One problem with Telehouse is that it is central to so much of our infrastructure, any resilience built-in elsewhere is only going to be able to cope with a fraction of the total traffic. People who buy cheap, non-resilient services get what they pay for – traffic dropped at the first sign of a problem.
When I helped design BT’s data centre infrastructure (many years ago now) absolutely EVERYTHING was resilient (including dual power feeds + standby generators). Unfortunately, it was so expensive to implement (>£5M per datacentre – then – , plus servers etc…) that BT dumped it and used a non-resilient solution. Oops!
very much doubt it.
If you worked at BT designing DC’s then you’d know resilience is the foundation of design.
Space in DC’s is sold based on parameters like cooling, power, security, ISP’s and resilience of the above.
it is perfectly reasonable to build a DC with no resilience if all systems within are replicated on a geographically separated mirror DC, thereby providing the resilience.
Sounds like a total troll KR
As a vast proportion of UK internet traffic passes through LINX, it is interesting to see what they have said.
This is what LINX has as networks spanning London:
The first event, yesterday, had this outcome from LINX:
Given they reach over 2Tbps through LINX in London, a loss of 20% is pretty big. The blip is visible on their traffic stats (it has moved off the daily stats, onto the weekly ones now), and seems to have affected their Juniper LAN.
https://www.linx.net/london/tech-info-help/traffic-stats
I think the TeleCity location houses one of LINX’s core locations, as does Telehouse North. It’ll be interesting to see what they say for today. The traffic graphs don’t show a blip at 7am, but the “Extreme LAN” is currently running a lot lower than the last few days.
http://www.telehouse.net/resources/blog/march-2016/the-cost-of-unplanned-outages
Marvellous….
“With more than 25 years of expertise, Telehouse’s high quality data centres are fully redundant. Furthermore as the only UK data centre campus with its own privately owned 132KV Grid Substation clients are provided with a secure and reliable power system to ensure they never go offline.”
I’m bang in the middle of London, and my connection (Xilo/Uno FTTC) was perfectly fine throughout the BT crisis.
My connection uses the Daisy Plc. network from exchange onwards.
Feeling very smug just now 🙂
I’m honestly not sure if this affected me (using Three mobile) or not. There were a couple of sites where I got the “Pale Moon can’t find the server” message, but they’re the sort of ten cents a month cowboy hosted sites that commit suicide every time someone starts their car in Clay County, Kansas, anyway.
Zero issues via TalkTalk Business.
Seems to have impacted those taking handovers from BT through affected kit and those whose transit and peering is centred around the affected kit.
It is a little worrying that this had the impact it did, however symptomatic I suspect in many cases of an industry where customers demand cheap and good, and ISPs do their best to achieve the first while papering over cracks to give the impression of the second.
Sound analysis I reckon.
What surprised me most, looking back at the first one, is how the disruption persisted long after the immediate problem ended.
What was then a surprise for the second one was that the disruption seemed to be over much sooner, while the fault was fixed later.