UPDATE7 Major BT Fault Affecting UK Internet and Broadband Services

Multiple ISPs are reporting that a fault, which is most likely on BT’s national broadband network, is affecting Internet connectivity for a large number of people across the United Kingdom (i.e. disrupting access to online services, websites and slowing connection speeds).

The problem itself began after 8am this morning and does not appear to be affecting every ISP and area, although it is extremely widespread and right now none of the ISPs are able to provide a clear picture of the cause.

BT Service Status Update

We currently have a major broadband problem that is preventing webpages from loading for some customers. This is also stopping some users from logging into their email accounts via webmail. Our engineers are working to fix this as quickly as possible.

However two ISPs have pointed the finger towards a possible problem at a major London data centre, which may be suffering from a routing problem that is connected to a power failure (if it’s routing then a fix shouldn’t be far behind). Now here’s a selection of some ISP tweets, until we know more.

https://twitter.com/BTCare/status/755676292717088768

UPDATE 10:21am

According to Entanet, “The problem has been confirmed on BT’s side and they are currently investigating. We haven’t been provided anything relating to the cause as of yet, but we will update upon receipt of this information. Unfortunately the BT systems remain down preventing us from looking into individual line issues.”

UPDATE 10:43am

The official line from BTOpenreach is as follows: “We’re sorry that some BT and Plusnet customers are experiencing problems accessing some internet services this morning. This is due to power issues at one of our internet peering partners’ sites in London. Engineers are working to fix things as fast as possible.” Of course it’s not just BT and Plusnet customers being affected.

On the upside some traffic now appears to have been rerouted and customers are reporting an improvement, but there are still a lot of complaints coming in. We suspect this will be resolved fairly soon.

UPDATE 10:52am

The power failure appears to have occurred at London’s TeleCity data centre.

UPDATE 11:07am

A private report from TeleCity, which was sent to certain providers earlier today and seen by ISPreview.co.uk, suggests that there was a power failure with one of their Uninterruptible Power Supply (UPS) systems at 8/9 Harbour Exchange (LD8) in London.

The fault hit a key router (edge4-tch), although this was technically resolved at 9:15am. However the knock-on impact for ISPs like BT and their clients (consumers, businesses and other ISPs etc.) can often take a little bit longer to resolve (computer networks are complicated animals).

Connections should be slowly getting back to normal.

UPDATE 11:57am

BT informs us that the problem is affecting around 10% of Internet usage on their network. So essentially, when users go to certain websites which are routed via the faulty part of the network, around 10% of their attempts to connect to the website they want to go to may fail. However, as posted above, normal services are slowly being restored.

UPDATE 1:17pm

The London Internet Exchange (LINX) has furnished us with the following statement to help clarify their role in events: “This morning between 07:55 BS and 08:17 BST, one of the Datacentres that houses equipment for The London Internet Exchange (LINX) experienced a partial power outage. This affected only one of a number of internet peering nodes that LINX operates at the facility, and service was fully restored on the LINX network at 09:15 BST. Several reports claim that this outage affected British Telecom and their services at LINX. While several networks connected to LINX were indeed affected, BT was not one of them. LINX provide two fully redundant platforms to offer better resilience to the UK’s Internet infrastructure, the second platform was not affected by this power outage.”

UPDATE 21st July 2016

A very similar fault has struck again this morning (read our coverage).