Major Network Outage Strikes Broadband ISP Virgin Media UK UPDATE10

Customers of broadband ISP Virgin Media (VMO2) are reporting a major UK network outage this morning (technically it’s been two outages), which has caused connectivity problems and appears to have disrupted their website and DNS servers too. Some reports indicate that their TV (IPTV), mobile and phone services have also been hit.

As usual in these situations, Virgin Media’s general Service Status page has gone offline with an error (this always happens with them, it’s a source of continued annoyance) and, at the time of writing, even their backup National Outage Page fails to load. Meanwhile, the operator has yet to post a status update to their social media channels, which is despite there being issues throughout the night.

According to Down Detector, the operator first suffered a large disruption to service just after 1am last night that trigged tens of thousands of complaints, but this appeared to have settled back down by around 4am. Unfortunately, the same thing happened again this morning at around 6:30am and has triggered a huge volume of complaints, which is most likely because more people are awake to experience it.

A quick look at internet traffic across Cloudflare’s global content delivery network (via Cloudflare Radar) also appears to show that they’re “tracking a complete internet outage at Virgin Media (AS5089) in the United Kingdom … Virgin Media DNS also unavailable.” The issue may also be extending to some of the operator’s business connectivity clients.

At the time of writing we’re still gathering data, but it appears as if the physical broadband connection for many customers is still technically live, yet there seems to be an issue with their routing/peering/DNS (Domain Name Servers) that is disrupting the internet connectivity side (it’s sporadic). We suspect it’s more routing orientated, as changing the DNS server to a third party (OpenDNS, Google Public DNS etc.) doesn’t seem to help.

We are currently trying to get a statement from Virgin Media and will keep this article updated. But we’ve not seen a network outage this large for a few years.

UPDATE 8:11am

Cisco’s monitoring of Border Gateway Protocol (BGP) events, which is a protocol that helps to link the internet together by exchanging routing information with Autonomous Systems (such as those run by your ISP), has also picked up the outage (here) – this has affected 239 prefixes.

UPDATE 8:25am

Sources inform us that the network outage may also be impacting Virgin Media’s internal support lines and systems. But it only takes one admin with a mobile phone to update their social media feed, yet so far, they haven’t even managed to do that. The level of communication from the provider on this is shockingly bad.

UPDATE 8:30am

Finally, after several hours, we have a statement. But it’s as uninformative as you’d expect.

A VMO2 Spokesperson said:

“We’re aware of an issue that is affecting broadband services for Virgin Media customers as well as our contact centres. Our teams are currently working to identify and fix the problem as quickly as possible and we apologise to those customers affected.”

UPDATE 8:40am

Virgin Media’s website seems to be coming back to life, although comically their major outage page states: “There are no major network outages we’re currently aware of.” We are also seeing some signs of an improvement elsewhere, but if they’ve corrected the issue with their routing/peering, then it may take a bit of time to fully return for everybody.

UPDATE 9:57am

Arriving somewhat after the fact, VM Business has just issued a similar statement to their residential division: “We’re aware of an issue that is affecting broadband services for Virgin Media Business customers as well as our contact centres. Our teams are currently working to identify and fix the problem as quickly as possible and we apologise to those customers affected.”

We’re currently seeing many services return to normal.

UPDATE 11:39am

The issue is now formally resolved. A VM spokesperson said: “We’ve restored broadband services for customers but are closely monitoring the situation as our engineers continue to investigate. We apologise for any inconvenience caused.”

UPDATE 4:40pm

Another massive outage appears to have begun occurring after 4pm today and, once again, their website appears to be affected.

UPDATE 6:49am

The situation has started to improve again, although Virgin Media has also confirmed the return of the issue. A spokesperson said: “Unfortunately, we have seen a repeat of an earlier issue which is causing intermittent broadband connectivity problems for some Virgin Media customers. We apologise again to those customers impacted and our teams are continuing to work flat out to find the root cause of the problem and fix it.”

UPDATE 5th April 2023 @ 9:09am

We’re still seeing some reports of issues, but only at a much lower level. Otherwise, we note that Virgin Media, working alongside Juniper engineers, apparently had to reset some hardware (FPC) at a number of its sites (e.g. Brentford, Northampton etc.) after they reported errors.

Interestingly, the operator has also applied further DDoS mitigation at their Brentford Backbone-3, although we don’t know for sure if the incident itself was related to an attack or an issue of maintenance/configuration. The remaining Backbone routers at Northampton, Birmingham, Manchester and Leeds have also had the mitigation applied.

UPDATE 5th April 2023 @ 11:56am

We asked Ofcom if they were aware of the outage and involvement of a DDoS attack. A spokesperson for the regulator told us: “We are aware that Virgin Media has been experiencing problems with its network. We are in contact with the company to establish the scale and cause of the problem as soon as possible.”