Home
 » ISP News » 
Sponsored

UPDATE2 2nd Major Power Fault Affects UK ISPs, BT and Internet Services

Thursday, July 21st, 2016 (9:27 am) - Score 4,283

After yesterday’s major disruption (here) we’re loathed to report that this morning appears to have started off in much the same way, with broadband ISPs from across the UK all reporting similar Internet disruption as a result of another possible power failure at a major London data centre.

Several ISPs are already pointing the finger at a power failure stemming from Telehouse North (specifically TFM10 as all the other kit is still powered). Meanwhile another ISP said they’ve lost all their London resilient interconnects to the BTWholesale DSL platform, although this has failed over to Manchester and so the impact was minimal.

The problem itself began after 7am this morning and once again it is affecting routing / DNS connectivity for multiple ISPs (i.e. you might have trouble accessing some websites and Internet services), albeit mostly those that use BT’s platform as a lot of ISPs connect through them. At the time of writing Telehouse are still working to restore power.

We should point out that yesterday’s fault stemmed from TeleCity, not Telehouse (i.e. different site), although it’s easy to get the two confused. No doubt questions will be asked about why this has happened twice in a row, albeit at different sites. Here’s a summary of some ISP updates.

https://twitter.com/fastbroadband/status/756036118215262208

UPDATE 10:06am

We’re getting tentative reports that the situation is now being resolved, although like yesterday it could take a couple of hours before everything returns to normal. In some cases you might be able to get rid of the problem by power cycling (rebooting) your router, but don’t do this repeatedly as it may harm your service speeds.

UPDATE 12:29pm

Telehouse Europe has finally issued a brief statement on the matter: “We are aware that there has been an issue with the tripping of a circuit breaker within Telehouse North that has affected a specific and limited group of customers within the building. The problem has been investigated and the solution identified. Our engineers are working with our customers on the resolution right now. We will release updates in due course.”

Some equipment will need to be replaced, although most ISPs have already managed to move traffic around the problem area.

Leave a Comment
26 Responses
  1. Avatar wirelesspacman says:

    It’s the wrong kind of summer! πŸ™‚

    1. Mark Jackson Mark Jackson says:

      Wrong kind of wind, wrong kind of rain, wrong kind of summer.. when is it the right kind of anything anymore? πŸ™‚

    2. Avatar dragoneast says:

      Perhaps the UK has the wrong kind of people?

    3. Avatar wirelesspacman says:

      4th – 6th August, Worcester Beer Fesitival…

      … most DEFINITELY the right kind of beer!! πŸ™‚

    4. Mark Jackson Mark Jackson says:

      ..unless it’s watered down πŸ˜‰ .

    5. Avatar wirelesspacman says:

      I will aim to undertake some rigorous testing then and let you know πŸ™‚

  2. Avatar danni says:

    probably because the network needs a new revamp but they rather bodge it instead of fixing it but say no more its British Telecom and the cowboys

  3. Avatar DTMark says:

    Does this, or yesterday’s issue, affect Virgin Media?

    1. Avatar GNewton says:

      No, it doesn’t, as far I know. I feel sorry for those who have to use BT’s network.

    2. Avatar DTMark says:

      Intrigued by Zen’s comment: “A BT outage is affecting all ISPs.”

      It didn’t seem to impact EE 4G yesterday nor today. All working perfectly here.

      Assuming that VM’s network is routed differently, it may not affect them either.

      So it’s not “all ISPs”, is it.

      But it does go to show that when it actually comes down to it and regardless of how much OFCOM might crow about success creating a competitive market, parts of the routing are the same, the BTW backhaul is the same for most, the cables to the property are the same, the engineers are the same..

      There isn’t really much in the way of real competition at all as they share the same bottlenecks and failure points and so the customer has remarkably little choice.

    3. Avatar Curious says:

      Since when did a mobile operator become an ISP?

    4. Avatar DTMark says:

      Why is it not an ISP?

      We’ve used MBB for our office and home connection for about 6 years now. It’s the fastest and seemingly most resilient connection available here.

      Fixed line only has relatively slow VDSL to offer, and there’s no cable.

    5. Avatar wirelesspacman says:

      @ Curious

      Since the invention of the smart phone

  4. Avatar Optimist says:

    This is a timely reminder of the lack of resilience by routing virtually all the UK’s internet traffic via one central hub. ISPs should be establishing alternatives in case of a major failure in London Docklands.

    1. Avatar Piers says:

      The lack of resilience is more individual companies investment rather than not being available. We interconnect with BT (and other suppliers) in both London and Manchester datacentres so all our customers automatically moved over to Manchester within 10 minutes of the failure this morning. The cost of having that capability and bandwidth is very high however and so a lot of ISPs fail to put in that level of investment due to costs being driven down by customers. You are right that a single datacentre is always a point of failure, no matter, how rare and so using multiple datacentres in multiple cities is paramount. Something any customers should consider when choosing a supplier.

    2. Avatar MikeW says:

      That resilience is important, and expensive. Like anything that requires a subscriber to pay more per month, it isn’t a feature that sits high on people’s priority list until just after they’ve been hit by it.

      One problem with Telehouse is that it is central to so much of our infrastructure, any resilience built-in elsewhere is only going to be able to cope with a fraction of the total traffic. People who buy cheap, non-resilient services get what they pay for – traffic dropped at the first sign of a problem.

  5. Avatar KR says:

    When I helped design BT’s data centre infrastructure (many years ago now) absolutely EVERYTHING was resilient (including dual power feeds + standby generators). Unfortunately, it was so expensive to implement (>Β£5M per datacentre – then – , plus servers etc…) that BT dumped it and used a non-resilient solution. Oops!

    1. Avatar Chris P says:

      very much doubt it.
      If you worked at BT designing DC’s then you’d know resilience is the foundation of design.
      Space in DC’s is sold based on parameters like cooling, power, security, ISP’s and resilience of the above.
      it is perfectly reasonable to build a DC with no resilience if all systems within are replicated on a geographically separated mirror DC, thereby providing the resilience.

    2. Avatar FibreFred says:

      Sounds like a total troll KR

  6. Avatar MikeW says:

    As a vast proportion of UK internet traffic passes through LINX, it is interesting to see what they have said.

    This is what LINX has as networks spanning London:

    The LINX London Network consists of two separate high-performance Ethernet switching platforms installed across 11 locations. These are located within datacentre sites operated by Telehouse Europe, TelecityGroup, Interxion, Equinix and Digital Realty in a metropolitan network spanning 64km.

    Switches from different equipment vendors, Extreme Networks and Juniper, are deployed in two diverse networks to provide an extra level of fault-tolerance, the logic being that both systems shouldn’t develop the same fault at the same time.

    Therefore, at least two switches are installed (one from each vendor) in every LINX location, with each site interconnected by multiple 10 gigabit Ethernet circuits (across dark fibre) to form the two physically separate backbone rings.

    The first event, yesterday, had this outcome from LINX:

    This morning between 07:55 and 08:17 BST there was a loss of power at Telecity Harbour Exchange, one of the datacentres that houses equipment for the London Internet Exchange (LINX). This power failure was not specific to LINX; other operators in the same building also lost power to their equipment.

    Power was restored to all our equipment by 08:17 BST, apart from a single device. Members with redundant connections will have found their service return to normal at this time. Power was fully restored at 09:15, which is the point we consider the incident window to be closed.

    LINX operates two fully redundant peering LANs in London, spread throughout 11 datacentres. Only one of the two LANs was affected by this incident. 80% of the traffic levels at the start of the incident continued to pass through LINX unaffected.

    Given they reach over 2Tbps through LINX in London, a loss of 20% is pretty big. The blip is visible on their traffic stats (it has moved off the daily stats, onto the weekly ones now), and seems to have affected their Juniper LAN.
    https://www.linx.net/london/tech-info-help/traffic-stats

    I think the TeleCity location houses one of LINX’s core locations, as does Telehouse North. It’ll be interesting to see what they say for today. The traffic graphs don’t show a blip at 7am, but the “Extreme LAN” is currently running a lot lower than the last few days.

    1. Avatar Kev says:

      Marvellous….
      “With more than 25 years of expertise, Telehouse’s high quality data centres are fully redundant. Furthermore as the only UK data centre campus with its own privately owned 132KV Grid Substation clients are provided with a secure and reliable power system to ensure they never go offline.”

  7. Avatar TTT says:

    I’m bang in the middle of London, and my connection (Xilo/Uno FTTC) was perfectly fine throughout the BT crisis.
    My connection uses the Daisy Plc. network from exchange onwards.
    Feeling very smug just now πŸ™‚

  8. Avatar sentup.custard says:

    I’m honestly not sure if this affected me (using Three mobile) or not. There were a couple of sites where I got the “Pale Moon can’t find the server” message, but they’re the sort of ten cents a month cowboy hosted sites that commit suicide every time someone starts their car in Clay County, Kansas, anyway.

  9. Avatar Ignition says:

    Zero issues via TalkTalk Business.

    Seems to have impacted those taking handovers from BT through affected kit and those whose transit and peering is centred around the affected kit.

    It is a little worrying that this had the impact it did, however symptomatic I suspect in many cases of an industry where customers demand cheap and good, and ISPs do their best to achieve the first while papering over cracks to give the impression of the second.

    1. Avatar MikeW says:

      Sound analysis I reckon.

      What surprised me most, looking back at the first one, is how the disruption persisted long after the immediate problem ended.

      What was then a surprise for the second one was that the disruption seemed to be over much sooner, while the fault was fixed later.

Comments are closed.

Comments RSS Feed

Javascript must be enabled to post (most browsers do this automatically)

Privacy Notice: Please note that news comments are anonymous, which means that we do NOT require you to enter any real personal details to post a message. By clicking to submit a post you agree to storing your comment content, display name, IP, email and / or website details in our database, for as long as the post remains live.

Only the submitted name and comment will be displayed in public, while the rest will be kept private (we will never share this outside of ISPreview, regardless of whether the data is real or fake). This comment system uses submitted IP, email and website address data to spot abuse and spammers. All data is transferred via an encrypted (https secure) session.

NOTE 1: Sometimes your comment might not appear immediately due to site cache (this is cleared every few hours) or it may be caught by automated moderation / anti-spam.

NOTE 2: Comments that break our rules, spam, troll or post via known fake IP/proxy servers may be blocked or removed.
Cheapest Superfast ISPs
  • Vodafone £22.00
    Speed 35Mbps, Unlimited
    Gift: None
  • Hyperoptic £22.00
    Speed 50Mbps, Unlimited
    Gift: Promo Code: HYPER21
  • Onestream £22.50 (*27.99)
    Speed 45Mbps, Unlimited
    Gift: None
  • xln telecom £22.74 (*47.94)
    Speed 66Mbps, Unlimited
    Gift: None
  • Plusnet £22.99 (*36.52)
    Speed 36Mbps, Unlimited
    Gift: Β£65 Reward Card
Large Availability | View All
Cheapest Ultrafast ISPs
  • Vodafone £26.00
    Speed: 100Mbps, Unlimited
    Gift: None
  • Virgin Media £27.99 (*51.00)
    Speed: 108Mbps, Unlimited
    Gift: None
  • Hyperoptic £29.00 (*35.00)
    Speed: 150Mbps, Unlimited
    Gift: Promo Code: HYPER21
  • TalkTalk £32.00 (*39.95)
    Speed: 145Mbps, Unlimited
    Gift: None
  • Giganet £35.00
    Speed: 200Mbps, Unlimited
    Gift: None
Large Availability | View All
The Top 20 Category Tags
  1. FTTP (3064)
  2. BT (2887)
  3. FTTC (1836)
  4. Building Digital UK (1819)
  5. Politics (1783)
  6. Openreach (1711)
  7. Business (1547)
  8. FTTH (1352)
  9. Mobile Broadband (1349)
  10. Statistics (1323)
  11. 4G (1160)
  12. Fibre Optic (1112)
  13. Wireless Internet (1083)
  14. Virgin Media (1076)
  15. Ofcom Regulation (1069)
  16. EE (767)
  17. Vodafone (749)
  18. TalkTalk (717)
  19. Sky Broadband (704)
  20. 5G (633)
Promotion
Helpful ISP Guides and Tips
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
Sponsored

Copyright © 1999 to Present - ISPreview.co.uk - All Rights Reserved - Terms , Privacy and Cookie Policy , Links , Website Rules , Contact