UPDATE3 Major Fibre Cable Break Causes Broadband Routing Woes for UK ISPs

A major fibre optic cable breakdown triggered serious routing problems for a multitude of different UK internet providers (e.g. Sky Broadband, O2, BT etc.) over the bank holiday weekend, which prevented broadband customers from reliably connecting to certain websites and other online services / servers.

Little is known about the incident itself, which appears to have started during Saturday Sunday (26th August 2012) night and continued on until the very early hours of this morning (experiences vary between providers), except for a bunch of generally vague service status updates that cropped up from different ISPs.

O2Wholesale Service Status Update (27/08/2012 16:08)

“The issue is related to major fibre cable breakdown. It is escalated and we have no any estimation when the issue will be resolved. Because of the nature of the issue which is outside of our network, we will update you as soon as we have more information.”

Thankfully most providers reacted quickly and began routing traffic around the problem, at least until it can be resolved, although some providers sadly took a little longer to react. The bank holiday period probably didn’t help matters.

The cause of all this remains unconfirmed, although a post on both the BE Usergroup and Sony’s Call of Duty Community Forum suggest that it was trigged by a sub-sea cable break between the UK and Netherlands section of Interoute’s transmission network. This appears to be supported by some evidence from European internet traffic monitoring sites, which showed similar problems.

As it stands most ISPs inform us that they are still “investigating” and cannot yet confirm the exact cause. We will continue to hunt for answers.

UPDATE 12:21pm

ISP BE Broadband (O2) just reported that their “network team worked with the transit providers and a traffic reroute has been put in place“, which they claim should bring “an improvement until the fibre cut causing unreachable destinations is repaired“.

UPDATE 1:29pm

After some prompting we’ve been able to get a more detailed statement on the incident from Interoute.

Matthew Finnie, CTO at Interoute, said:

“On Monday 27th August 2012 at 02.57 UTC, one of Interoute’s European cable systems was affected by a subsea cable break approximately 120km off the UK shore and 100km off the Netherlands shore.

A specialist cable marine crew and vessel have been engaged. The timeline for a full system fix is currently pending. The cause of the fibre break has not yet been fully determined. However, current information suggests that the culprit was a ship’s anchor.

Approximately 400Gbps of services have been successfully re-routed and handed back into active service since the break occurred. In the next 48 hours, Interoute aims to have 75-80% of the total unprotected services back up and running. None of Interoute’s protected services, e.g. Internet, corporate connectivity, hosting or video have been affected.

We apologise for any inconvenience caused by this cable break.”

As above, most ISPs appear to confirm a similar re-routing of their data traffic.

UPDATE 29th August 2012

Some readers were understandably a little confused by yesterday’s Interoute statement, which suggested that the problem began on Monday morning. As stated in our article above, reports of related problems first began to surface on Sunday evening.

Today O2 has given us a statement which appears to confirm that the problem did actually begin on Sunday and also apologises for the fact that they took longer than most to resolve the issue.

An O2 Spokesperson told ISPreview.co.uk:

“We can confirm that on Sunday 26 August, a damaged deep sea cable affected multiple ISPs which resulted in a number of O2 Broadband customers having difficulty accessing certain websites.

Our Broadband team worked with our internet transit supplier to re-route service which was completed by 19.00 hrs on Monday 27 August when normal service resumed for O2 customers.

We’d like to apologise to our customers for any inconvenience this issue may have caused. We will continue to monitor the service while repairs by the cable owners continue.”