ISP Zen Internet Sorry After 11k UK Users Hit by 8 Hour Broadband Outage

Popular ISP Zen Internet has apologised after 11,000 of its broadband customers (FTTC and ADSL etc.) were left without service for nearly 8 hours on Sunday (27th April 2014) after a fibre optic cable broke. Sadly the provider’s network then failed to transfer affected services over to alternative circuits.

In an apologetic message Zen’s Head of Network & Infrastructure, Robin Griffith, explained that the ISPs network is designed to transfer to alternative circuits in the case of a failure but this “did not operate as expected” during Sunday’s incident.

The broken fibre circuit has since been restored but Griffith noted that their investigations into the “failover issue” (fail to failover you might say) is on-going and apparently “points to an unusual set of conditions affecting the equipment involved“.

Robin Griffith said:

“The outage unfortunately impacted over 11,000 broadband customers nationwide, both Fibre and ADSL, many of whom were without service for nearly 8 hours. Although the issue has now been resolved, I’d like to apologise for the inconvenience this outage has caused for many of our customers.

The resiliency of our network is crucial in our ability to provide you with the service that you have come to expect from Zen. I would therefore like to assure you that we are continuing to investigate the unique set of conditions which caused the back-up service not to operate on this occasion.”

Suffice to say that such serious network-wide outages are not normal for Zen, which has earned a strong reputation for quality. Griffith also agreed that the ISP had failed to provide enough detail about the problem via their @zeninternet Twitter account and promised to improve matters going forward.

In addition, Zen has also published a brief overview that shows how the outage occurred and what they did to resolve the problem.

Zen’s Outage – Sunday 27 April

10.14 – We were alerted that there may be a potential problem via our standard monitoring processes.

10.27 – It became clear that the outage was on our Manchester Point of Presence (POP).

10.47 – The Service Desk Team confirmed that it was a localised issue to Zen’s network.

11.38 – The Network Engineering Team who were investigating the outage realised the scale and complexity of the incident and this was then escalated to the Network Operations Manager who in turn, informed myself.

13.00 – Failure of the back-up circuits to provide alternative service led the team to go on-site at our Manchester POP to carry out further investigations.

15.53 – The fibre break was localised and the suppliers directed to the point of failure. The engineers continued to investigate why the back-up service we have in place for rare outages such as this had not activated.

17.18 – A Supplier Engineer arrived at the POP and repaired the fibre break.

18.14 – Confirmation that our Broadband customers services had been restored.