Sponsored

Severe loss to Cloudflare targets from Virgin Media

Msh

Casual Member
Hey guys,

Hopefully you can assist me in getting more visibility into an issue I am having on Virgin Media. I am having daily (7 consecutive days now) severe packet loss to targets on Cloudflare. This includes 1.1.1.1 and a lot (seems random) of Cloudflare websites that go over the route with a Virgin core router at 212.250.14.2.

I have posted on the Virgin Media forums and I am getting a fair amount of agreement there.

The issue however is it's clear the there's a lack of severity. Maybe someone doesn't understand the impact of Cloudflare issues or are downplaying the loss.

I've also asked how I should be contacting support for this issue however have had no reply to that query.

Therefore, does anyone know:
- How do I properly escalate this issue? It is in need of urgently being looked into.
- What are the correct channels to escalate this? I have had zero luck myself. Would a switch to business broadband help me in the future?

Here's my packet loss over the last 8 days (the metrics are averaged and smoothed to see the corolation between 1.1.1.1 and the VM core router clearly). It seems to be related to peak hours.
U0rPkH7.png


Note, I have also contacted Cloudflare, but with no reply yet. Free customers get deprioritised unfortunately.

Thanks!
Msh
 

Buggerlugz

ULTIMATE Member
I think cloud-flare has been having issues due to the take-up of their new Warp app. Yesterday something was definitely flaky with Warp and its DNS server. Today seems like the problems resolved itself.
 

Msh

Casual Member
I think cloud-flare has been having issues due to the take-up of their new Warp app. Yesterday something was definitely flaky with Warp and its DNS server. Today seems like the problems resolved itself.
Unfortunately the loss directly correlates with loss on a VM router along the route, so I really doubt it's Cloudflare's side.
 

forid200

Casual Member
I've been seeing something similar for sometime - I use pfSense with my Virgin Media router in modem mode.

I don't believe this is a problem with Cloudflare, I've tested from other ISPs and they don't have this issue when pinging cloudflare. Something is going amiss with the routing between Cloudflare and VM. I've tried to raise with VM, but cannot get through to anyone who knows anything, just those few people reading off a script.

How are you graphing your data? I'm guessing that's Grafana? Looks awesome, thinking of doing something similar soon for a weekend project.
 

Msh

Casual Member
I've been seeing something similar for sometime - I use pfSense with my Virgin Media router in modem mode.

I don't believe this is a problem with Cloudflare, I've tested from other ISPs and they don't have this issue when pinging cloudflare. Something is going amiss with the routing between Cloudflare and VM. I've tried to raise with VM, but cannot get through to anyone who knows anything, just those few people reading off a script.

How are you graphing your data? I'm guessing that's Grafana? Looks awesome, thinking of doing something similar soon for a weekend project.
Oh the issue is absolutely not Cloudflare. I totally agree, it’s Virgin at fault some how. I’ve only reached out to cloudflare as I’m a bit more optimistic about getting them to apply pressure.

The data is collected with Telegraf and its ping plugin which is fed into InfluxDB, the data is then visualised in Grafana indeed. Hope that helps!

Msh
 

forid200

Casual Member
Oh the issue is absolutely not Cloudflare. I totally agree, it’s Virgin at fault some how. I’ve only reached out to cloudflare as I’m a bit more optimistic about getting them to apply pressure.

The data is collected with Telegraf and its ping plugin which is fed into InfluxDB, the data is then visualised in Grafana indeed. Hope that helps!

Msh
Thanks for the information, really useful to know. I'll be setting something up soon because Virgin media has been funny since March for me. Contract expires end of the year, I'll be giving it a real think. Speed is pretty decent with virgin media but reliability isn't great.

Please keep us up to date with what you hear back from Cloud flare, the guys at cloud flare are pretty amazing and I'm sure they'll be able to make more noise than us niche guys picking up a few ping losses.
 

forid200

Casual Member
Just an update - this issue now seems to be sorted for me. Packets are no longer getting dropped from my Virginmedia connection.

@Msh Can you confirm is this is also resolved for you?
 

forid200

Casual Member
Interesting - yesterday I had a few outages. I wonder if they made any changes since then.

1tEUYOG.png


What area code are you in? I'm in 22.
 

Msh

Casual Member
Interesting - yesterday I had a few outages. I wonder if they made any changes since then.

1tEUYOG.png


What area code are you in? I'm in 22.
7, but this is definitely nationwide.

It's interesting that Thursday and Friday were good after 15 days of issues.
 

TTJJ

ULTIMATE Member
This kind of thing seems to be fairly common with Virgin Media.

Their network core doesn’t seem to be very well managed. I used them for a while and whilst it’s fine for most consumers I’d notice occasional quirks and strange issues. The same for their 4G service at the time too.
 

forid200

Casual Member
This kind of thing seems to be fairly common with Virgin Media.

Their network core doesn’t seem to be very well managed. I used them for a while and whilst it’s fine for most consumers I’d notice occasional quirks and strange issues. The same for their 4G service at the time too.
Very common, routing messes up randomly. Same goes for other providers who rely on them for transit. Those often get affected to.
 

TTJJ

ULTIMATE Member
Despite having the super fast speeds it seemed the connection was worse than the FTTC which was significantly slower.

There were a lot of devices online but had the router in bridge mode and the router was quite a good one - never had any issues with the router itself. It was all very odd which is why I ended up leaving mid-contract as it seemed to just be a characteristic of the Virgin network.
 

TTJJ

ULTIMATE Member
After raising a complaint they were fairly reasonable about me leaving with no ETFs.
Couldn’t complain about the complaints team but their general customer service wasn’t in any way helpful as they didn’t really know anything about the network.
 

forid200

Casual Member
After raising a complaint they were fairly reasonable about me leaving with no ETFs.
Couldn’t complain about the complaints team but their general customer service wasn’t in any way helpful as they didn’t really know anything about the network.
Virgin media has a lot of problems, I'm only just figuring them out recently because it's getting worse...

  1. Weird routing problems
  2. Really crappy router that's underpowered and has known chipset flaws
  3. RFoG???!
  4. Over subscribed as hell, on a docsis network causes really big latency spikes. The graph I posted above was on a good day. Usually gaming is impossible for example
Right now I'm using my router in modem mode with my own custom pfsense firewall and ubiquiti APs. Only way I've managed to cope with this garbage.
 

Msh

Casual Member
Virgin media has a lot of problems, I'm only just figuring them out recently because it's getting worse...

  1. Weird routing problems
  2. Really crappy router that's underpowered and has known chipset flaws
  3. RFoG???!
  4. Over subscribed as hell, on a docsis network causes really big latency spikes. The graph I posted above was on a good day. Usually gaming is impossible for example
Right now I'm using my router in modem mode with my own custom pfsense firewall and ubiquiti APs. Only way I've managed to cope with this garbage.
I am actually a new customer (April?) as they rolled out here as a lightning area. I agree with the first 3 from my experience so far but I can't say I have had problems with latency deviations that cause gaming trouble.

It's irritating that the 30 Mbps DSL I had before would be far more useful to me when those issues to Cloudflare do happen.

But no matter how you look at it, you can't dismiss 550 Mbps a majority of the time, so I am here to stay and praying Openreach step up their FTTP rollouts (not optimistic)
 

forid200

Casual Member
I am actually a new customer (April?) as they rolled out here as a lightning area. I agree with the first 3 from my experience so far but I can't say I have had problems with latency deviations that cause gaming trouble.

It's irritating that the 30 Mbps DSL I had before would be far more useful to me when those issues to Cloudflare do happen.

But no matter how you look at it, you can't dismiss 550 Mbps a majority of the time, so I am here to stay and praying Openreach step up their FTTP rollouts (not optimistic)
I believe in my area upstream traffic is congested - not good for me when I use a lot of cloud services and do upload quite often. I've called their support many times, tell them that I'm on a wired connection and they still try sell me boosters. I've requested escalation to their 2nd line team and got zero response. Their support lack any brain cells to even mention anything past their script.

Virgin media forums are flooded with reports of high latency making gaming impossible in a lot of areas around the country. But of course this is shadowing all the problems people WFH now will face with RDP, VDI and VPNs.

This year virgin has had very regular outages and drop outs too, some were nationwide. I love having a lot of speed at my fingertips, but the reliability of virgin is scrappy to say the least. This stems from their core networking, use of RFoG, flawed puma chipset in their routers, and they apparently also lack any form of basic monitoring. Most times they will have no idea if an entire part of London goes down until people complain about it.

They had great potential, being owned by Liberty Global but the network quality has let them down.
 

Msh

Casual Member
I believe in my area upstream traffic is congested - not good for me when I use a lot of cloud services and do upload quite often. I've called their support many times, tell them that I'm on a wired connection and they still try sell me boosters. I've requested escalation to their 2nd line team and got zero response. Their support lack any brain cells to even mention anything past their script.

Virgin media forums are flooded with reports of high latency making gaming impossible in a lot of areas around the country. But of course this is shadowing all the problems people WFH now will face with RDP, VDI and VPNs.

This year virgin has had very regular outages and drop outs too, some were nationwide. I love having a lot of speed at my fingertips, but the reliability of virgin is scrappy to say the least. This stems from their core networking, use of RFoG, flawed puma chipset in their routers, and they apparently also lack any form of basic monitoring. Most times they will have no idea if an entire part of London goes down until people complain about it.

They had great potential, being owned by Liberty Global but the network quality has let them down.
I am not trying to defend VM, quite the opposite.

The lack of route to escalate is a real problem, and it's something I want to find more about with VM business, for me that may be worth it. The need for 1st line is totally sensible, read through their forums and it's full of people with crappy wifi more than anything else. But they need to know when they have to get further assistance.

Like you, I am a heavy internet user. I work as a site reliability engineer and frequently upload/download bulky objects.

The lack of monitoring point angers me a lot, why am I, a private user, providing vast amounts of data to them about an issue they should have seen before end-user issues even occurred.
 

forid200

Casual Member
I am not trying to defend VM, quite the opposite.

The lack of route to escalate is a real problem, and it's something I want to find more about with VM business, for me that may be worth it. The need for 1st line is totally sensible, read through their forums and it's full of people with crappy wifi more than anything else. But they need to know when they have to get further assistance.

Like you, I am a heavy internet user. I work as a site reliability engineer and frequently upload/download bulky objects.

The lack of monitoring point angers me a lot, why am I, a private user, providing vast amounts of data to them about an issue they should have seen before end-user issues even occurred.
I understand, not saying you're defending them. Just blowing off some steam really, everything going down hill from their core network, technology used, modem provided and their tech support who probably know less than your grandma all add up to terrible customer experience.

I'm a Network infrastructure engineer myself, I'm getting asked by friends why their virgin media 350meg connections 'feels' slower than the BT 70meg they had before. And there's no easy answer to give them. Unfortunately, virgin don't care, as long as they can use the higher bandwidth figures to get more customers and make more money.

I'm also looking forward to gigabit offerings but so far none available in my area, and (unfortunately) the ones like BT Fibre 900 aren't symmetrical. Would love to have Hyperoptic, from what I've seen/heard it's really solid.
 

Msh

Casual Member
I understand, not saying you're defending them. Just blowing off some steam really, everything going down hill from their core network, technology used, modem provided and their tech support who probably know less than your grandma all add up to terrible customer experience.

I'm a Network infrastructure engineer myself, I'm getting asked by friends why their virgin media 350meg connections 'feels' slower than the BT 70meg they had before. And there's no easy answer to give them. Unfortunately, virgin don't care, as long as they can use the higher bandwidth figures to get more customers and make more money.

I'm also looking forward to gigabit offerings but so far none available in my area, and (unfortunately) the ones like BT Fibre 900 aren't symmetrical. Would love to have Hyperoptic, from what I've seen/heard it's really solid.
Well, you're lucky to be in an Openreach fibre area :D at least the ratios beat Virgins and their tech stack does allow for it to continuously increase... not even sure why DOCSIS is a thing when you feed fibre into someones home, mad.

Like I said before, I am between VDSL at about 29/10, or VMs network. The choice should be far more one-sided than it is.

Back on the topic a little bit though, I have not had loss *yet* today. I do think they're finally getting on top of it. Wonder when it'll be fully functional. And I do wonder if it was pressure applied from Cloudflare that pushed it along.
 
Top
Promotion
Cheapest Superfast ISPs
  • NOW £22.00 (*32.00)
    Speed 36Mbps, Unlimited
    Gift: None
  • TalkTalk £22.00 (*29.95)
    Speed 38Mbps, Unlimited
    Gift: None
  • Vodafone £22.00 (*25.00)
    Speed 35Mbps, Unlimited
    Gift: None
  • Hyperoptic £22.00
    Speed 50Mbps, Unlimited
    Gift: Promo Code: HYPERSPRING
  • Plusnet £22.99 (*36.52)
    Speed 36Mbps, Unlimited
    Gift: £50 Reward Card
Large Availability | View All
Cheapest Ultrafast ISPs
  • Gigaclear £24.00 (*44.00)
    Speed: 100Mbps, Unlimited
    Gift: Offer Code: SPRUCE20
  • Vodafone £26.00 (*29.00)
    Speed: 100Mbps, Unlimited
    Gift: None
  • Virgin Media £28.00 (*44.00)
    Speed: 108Mbps, Unlimited
    Gift: None
  • Hyperoptic £29.00 (*35.00)
    Speed: 150Mbps, Unlimited
    Gift: Promo Code: HYPERSPRING
  • TalkTalk £29.95 (*39.95)
    Speed: 145Mbps, Unlimited
    Gift: None
Large Availability | View All
Helpful ISP Guides and Tips
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
Promotion
The Top 20 Category Tags
  1. FTTP (3203)
  2. BT (2919)
  3. FTTC (1848)
  4. Building Digital UK (1847)
  5. Politics (1826)
  6. Openreach (1744)
  7. Business (1587)
  8. Mobile Broadband (1371)
  9. FTTH (1355)
  10. Statistics (1345)
  11. 4G (1181)
  12. Fibre Optic (1127)
  13. Wireless Internet (1106)
  14. Virgin Media (1099)
  15. Ofcom Regulation (1094)
  16. EE (784)
  17. Vodafone (775)
  18. TalkTalk (734)
  19. Sky Broadband (713)
  20. 5G (664)
Sponsored

Copyright © 1999 to Present - ISPreview.co.uk - All Rights Reserved - Terms  ,  Privacy and Cookie Policy  ,  Links  ,  Website Rules