UPDATE Data Corruption Issue Hitting Broadband Users Near Edinburgh

Broadband users with connections that are based off BTOpenreach’s national network (e.g. BT, Sky Broadband, TalkTalk), specifically those who reside around the eastern side of Edinburgh in Scotland (East Lothian), appear to be suffering from a curious fault that is causing downloads to corrupt.

The problem was picked up on by Thinkbroadband last night, although it appears to have begun last Thursday (10th September 2015) and almost none of the big ISPs are fully reporting it on their service status pages; that’s despite the existence of several long discussion threads by Sky Broadband, BT and TalkTalk subscribers (here, here and here).

We had a quick look around the ISPs and so far the only provider of any scale that appears to be correctly reflecting the problem is Zen Internet, which has a rather jargon rich fault report (43147) for the relevant area. Most of the status updates don’t clarify the issue itself, except for this one in the middle.

Update No. 6 – 14/09/2015 18:02

This is running as a Parent case P1 incident IMT43198/15. The TSO engineer at Falkirk has cleaned fibres and re-seated cards after which the traffic was routed back to the original route.

Traffic can be seen flowing but the customer is still seeing issues, 3rd line support are still monitoring the quality of the service as the customer can download files but the data is showing as corrupt.

As a parallel action the operations team are ordering spare cards for the end of the routes (Edinburgh and Tranent) TSO engineers will be tasked once ordered. The vendor is now on the bridge providing support. Next update will be issued upon service restoration unless there should be a significant development.

At the time of writing the last update on Zen’s page was posted at 4am this morning, which states that the CPM card was changed but “to no avail” and that “traffic levels have died down so investigations are being hampered“. “The technical call will reconvene at 09:00 to investigate further. Root cause and estimated restoration time unknown at present,” said the update.

The fault appears to affect customers on Openreach’s “fibre broadband” (FTTC/P) network, although we suspect that it might be possible to circumvent it by using a Virtual Private Network (VPN); this will be slow unless you subscribe to a premium service. Detecting these kinds of corruption errors isn’t easy and it’s one of those problems that can take a while to track down and fix.

UPDATE 10:13am

Zen has posted another update just now, which states that “initial indications are that the issues seen are persisting.”

UPDATE 1:14pm

Apparently BTOpenreach have replaced a network facing card on the layer 2 switch in Tranent, which initially seemed to resolve the problem. Unfortunately the layer 2 switch crashed a little while later, which required Openreach to carry out a number of switch card reseats and a card replacement.

As it stands now the problem appears to have been resolved, although data traffic is still being monitored for errors.