Last week it emerged that cable broadband ISP Virgin Media UK had begun deploying a new firmware update v9.1.116.603 for their Hub 3.0 (SuperHub v3) wireless router in order to fix the long running latency bug (here) on the Intel Puma 6 chipset. Today we wanted to share a quick update.
As a quick recap. The CPU (processor) inside the modem component of the Hub 3.0 router was taking on too much work while processing network packets, which caused the chipset to run a high-priority maintenance task every few seconds. Owners noticed that this task was occupying the CPU a bit too much and causing momentary latency spikes (increases of 200 milliseconds+), including a little packet loss.
The problem was disruptive enough that it could affect some time sensitive internet applications, such as fast paced multiplayer games. After a painfully long wait Virgin Media finally appeared to be rolling out a fix for the problem last week, which among other things shifted some of the load away from the CPU (we think the AR9382 comms chip takes it on) and did a few other things to improve the situation.
Since then we’ve been trying to get a solid line from Virgin Media on the firmware and now have a bit more information. Firstly, the operator has confirmed that the firmware is effectively a short trial and not everybody has received it because testing is still underway. Based on the pace of past updates we believe they should be able to complete the rollout to everybody by the end of April 2018, at the latest, but this is not confirmed.
A Spokesperson for Virgin Media told ISPreview.co.uk:
“New firmware for the Hub 3.0 is being deployed amongst a trial base. Further rollout will commence soon after further testing has taken place to address any unforeseen issues.”
At this stage it’s still unclear whether the new firmware only resolves part of the latency issue, such as with DNS lookups (a big chunk of these were lost with the current bug), which would mean that the spike could still be present in TCP/UDP, albeit now hidden from certain common tests (ICMP / Ping / BQM testing graphs etc.).
Feedback from those who have received the update appears to be mixed. Some report a much smoother experience in their multiplayer games, while others have found that the spikes are still occurring (games that shift bigger packets of data may be more affected); albeit still a bit improved from before the update.
Ultimately the Puma 6 is a fairly weak single-core CPU and there may be little that either Intel, ARRIS (Hub manufacturer) or Liberty Global can do to completely resolve the problem. A faster or multi-core CPU would have been better able to cope with the underlying issue but we won’t see that until a future Hub 4.0 launches (possibly something like the ARRIS TG3442). In the meantime Virgin’s partial fix will have to suffice.
We should point out that this bug is not unique to Virgin Media, it affects a variety of other routers with the Intel Puma 6 inside. Parent company Liberty Global chose the kit.
UPDATE 4th June 2018
Some of you have been asking for an update. So far the new firmware (e.g. v9.1.116.603 and .608) is supposed to have been deployed to both trial users and owners of the Hub 3.0 on Virgin Media’s top broadband tier (currently 350Mbps or 300Mbps before that), although Hub 3.0 owners on slower tiers have been left to wait.
A spokesperson told ISPreview.co.uk, “Further firmware updates are being worked on and we’re hoping to roll these out as soon as possible (testing permitted).” Sadly they couldn’t give a clearer indication than that.
UPDATE 14th August 2018
Intel seems to have finally coughed up to the related security flaw (here).
UPDATE 22nd August 2018
After much prompting we’ve finally heard back from Virgin Media on the status of their Hub 3.0 latency fix. The operator notes that their .608 firmware code has now been rolled out to around 90% of their base and they’ll push the upgrade to the remaining units in the “coming weeks“.
Virgin added that there have been no major performance issues identified in trials and pilot, although a small volume of post-rollout issues have been reported to their CPE (router) team and are being assessed at present.
I’m in the trial even though I wasn’t told I would be. I feel it may have been wasted on me as far as testing it while gaming as I won’t be home for the next few weeks.
Over on the VM forum thread there were a few people asking about gaming specifics with the new update but no real concrete answers just people saying they “read” it hasn’t made a difference. Just your standard vague and borderline useless stuff.
One thing I can be 100% certain of is that my BQM has improved:
https://www.thinkbroadband.com/broadband/monitoring/quality/share/6a115fd02598180be0435242919679cc7ac8eaee
Wish I could actually do some proper testing.
Before the latest update:
https://www.thinkbroadband.com/broadband/monitoring/quality/share/0300fe2bb91e2dfee7fa3156581041b567f2d438-12-02-2018
My old superhub (1st generation) has just been replaced with a SH3 – about 4 days ago.
I haven’t done any internet performance testing yet, but the hub web interface is SLLOOOWWWW and buggy.
1) takes ages to populate the list of connected devices (sometimes the spinning wheel doesn’t go away – so you have to hit ‘refresh’ to get the list.
2) setting a mac filter involves (what should be) a nice graphical interface for setting the filter times. You can click to add a filter, then drag it to move it, or drag the handles to move just the start or end time. Only the pointer icons (hand or double-headed arrow) don’t correctly show what will happen – sometimes with the double-headed arrow you’ll move the entire filter, sometimes you’ll drag the start or end time.
3) clicking apply changes sometimes just refreshes the screen, sometimes causes the full ‘applying changes’ popup and background animation.
4) having applied the changes, I then click on ‘home’ and get warned the changes have not been saved. clicking the new ‘apply changes’ button doesn’t appear to have any effect, cos clicking ‘home’ again causes the same warning.
5) the security timeout is very short – if you don’t work quick you get logged out.
Have to say, I’m not overly impressed.
… further to my post:
I have software version 9.1.116V
I was a beta testing on both Superhub 3 and Superhub 2, we got the hardware a year before public launch, and these issues were apparent. Many of the more technically minded people on the beta forum mentioned this stuff to the staff there, but it either fell on deaf ears, or the forum staff would just shrug and say there was nothing they could do.
Would be more beneficial if they sorted the horrendous WiFi signal around the home to be honest
Firstly sort out the physical environment in your home before you blame your router….
it is royally shit. Even online reviews say so. Go get abranded router and put this netgear rubbish into modem mode
From a business point of view, where do companies who’ve integrated the affected Puma 6 choosers into their products actually stand?
Is there scope for these companies to take legal action against Intel for essentially providing an inherently flawed product? As it seems they’re a bit prone to that of late – the last decade or so.
Also, if this issue was apparent when the hardware was being tested 1+ years before rollout, what’s the point in a trial if you use it anyway?
I suspect that’s one you’d have to test in a court of law, as some have already discussed. Personally I’m not a fan of suing CPU makers for chip flaws that they didn’t know about until after the silicon shipped. CPUs are so ridiculously complicated that I can’t see how such issues could be completely avoided. The caveat is I don’t know whether this logical applies in the case of Puma 6.
Humans are not perfect and engineering CPUs will thus never be perfect, you simply cannot predict precisely what might crop in every single eventuality until these things enter the mainstream. They also have to be mixed with other hardware and software, which the CPU maker won’t see until after the fact. Expecting perfection is unrealistic.
However your point about the issue being spotted in the pre-release stage on the final router design itself is more valid. I’d guess Liberty Global had already committed to a significant order by then and so the bug was not considered significant enough to prevent launch. Either way this raises questions about their approach to testing new kit and dependency on a single model, which might benefit from being internally reviewed.
I am on the trial. It fixes the UDP packet loss. Latency is slightly better, but still not anywhere close to fixed.
What’s the version of firmware that’s being tested? Anyone know? I use my superhub 3 in modem mode only and wonder if I can check to see what I should be running.
Just realised it’s at the start of the article 🙂
Can someone dumb thus down for an ordinary layman please. I have no idea what this means (ps: I am a VM 100mb BB customer)
I have had a running battle with virgin over the last year with
Packet loss and high latency – All of which has fallen on deaf ears and according to their systems wasn’t happening, even though I had all the graphs and line stats to prove the issues – their technical support & customer support was absolutely awful – So after 9 years as a customer I have left and gone to plusnet which has been faultless since plugging in.
The fix has improved gaming latency a little but my router is randomly suffering from massive packet loss since the firmware release. I have to check regularly and do a reset of my router to get it running again with no issues but this may last a day or so before it starts again.
https://www.thinkbroadband.com/broadband/monitoring/quality/share/c73fb4fb1e994254ceaf4fe5d5655b82d8c8f990-23-02-2018
Does putting the superhub into modem mode and using a third party router circumvent this problem? Or does the superhub fail at being a straight forward modem too?
Modem mode sucks as well. Chipset problem is underlying hardware issue that causes the fault in both Router and Modem mode
Ah that sucks. Was hoping that as a modem has a lot less work to do than a router that the chipset overload would be negated.
Hi all. Is there a conclusion on if this trail has been concluded and if so, is there a mass rollout?
I’m getting 30ms Latency, which seems twice that of my parents BT circuit.
Thanks!
Is there anyway of getting hold of a patch / new firmware i can install?
Still waiting for new firmware, virgin sort your rubbish superhub 3 out already, the amount of time you have known about the issues is criminal. Packet loss an latency issues are still common, at the end of my contract ill be off. How they ship these to users still with these issues is ridiculous.
Reading this is enough to make me leave VM after being a CabelTel/NTL/VM customer for over 20 years.
My hobby is simracing which is extremely latency and quality dependent – it’s not just about winning, having my car disappear from track whilst in close proximity to other racers is extremely disconcerting for them as they don’t know where I’m going to reappear.
How do we put pressure on VM/Liberty Media to sort this out, or can I only vote by taking my £1200/year elsewhere?
Thanks for taking the time to post this. Very informative. I’ve been from Hub 3.0 back to SH2AC and now have to have 3.0 to go for a higher package. Kept on to my SH2AC in case I have issues. Swapped over yesterday and even though in MODEM mode with my trusty ASUS RT-AC87U – it feels more laggy on the internet with the 3.0 🙁 – The Virgin Media Hub I have is VMDG505/TG2492LG-VM on 9.1.116V firmware
Just got an SH3. Rubbish.
BUT…
Fast.
So I am running a TP-link wifi router piggybacking the SH3.
Results are promising.
Speed is 200+ wired, and up to about 90 upstairs on WiFi.
A huge speed upgrade on the SH1 we were using before.
Virgin may be a bit rubbish, but there is no way I’d get those speeds from BT.
It is just a pity that one has to add some extra kit (a decent router!) and know a bit about networking too.
Game lag? Dunno.
Not everyone uses the internet for games!
They just swapped mine working SH2 to Hub3 during broadband testing. I regret that I didn’t research this before. I will try to get HUB2 back from virgin or buy one of eBay but would anybody comment/ confirm packet loss and high latency in my case?
https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/a57afe3664fa70f07a163e9b436df53dfe30898c-31-05-2018.png
Thanks.
Since moving to the Super Hub 3.0, as with everyone else on here and the Virgin community forums, I’ve had lots of issues with drop out. I too wish I could get my previous hub back and get rid of this 3.0. If they don’t fix the issue or issue a new router, I’ll be forced to reduce my internet speed and go to a new provider. Sad, but no choice, I’m afraid.
Well something is happening at VM with this. My router keeps flip flopping between being a Virgin Media branded device and a UPC branded device (the Irish version of the network) so clearly someone is updating devices remotely – and I’m getting continual device reboots becuase of it.
It’s particularly annoying that the router takes some 8 to 10 minutes to come back online when it occurs!
basically they trucking me for someones paycheck….
Great lady who sold me the lemon modem …. shame she was selling a lemon… nasty one
East Europe….how u feel being DdOSED back, Virgin gave the OKAY to protect myself… said they nothing they can do!!! Great service info or Jail time
(BigNutz- BigPeePee csgo)
Vm engineer came round and fixed my wireless drop outs. Superhub 3 in modem mode with ASUS router and merlin firmware. Just Changed wireless mode auto to N only. Performance not as good as my previous superhub, but at least it now works across multiple devices. Maybe useful to someone out there.
I’ve been following this issue for about a year now. Our SuperHub 3 is running the new firmware:
Software version : 9.1.116.608
However as you can see the packet loss between client and router is shocking. As people have rightly pointed out, this is a router issue not an issue with Internet speed:
Reply from 192.168.0.1: bytes=32 time=273ms TTL=64
Reply from 192.168.0.1: bytes=32 time=7ms TTL=64
Request timed out.
Request timed out.
Request timed out.
Reply from 192.168.0.1: bytes=32 time=6ms TTL=64
Request timed out.
Request timed out.
Request timed out.
Reply from 192.168.0.1: bytes=32 time=22ms TTL=64
Now and again, you’ll see a decent run of responses with no timeouts, but every other couple of pings will have a ridiculously high response time:
Reply from 192.168.0.1: bytes=32 time=221ms TTL=64
Reply from 192.168.0.1: bytes=32 time=6ms TTL=64
Reply from 192.168.0.1: bytes=32 time=2214ms TTL=64
Reply from 192.168.0.1: bytes=32 time=6ms TTL=64
Reply from 192.168.0.1: bytes=32 time=5ms TTL=64
Reply from 192.168.0.1: bytes=32 time=2029ms TTL=64
Reply from 192.168.0.1: bytes=32 time=7ms TTL=64
Reply from 192.168.0.1: bytes=32 time=5ms TTL=64
Reply from 192.168.0.1: bytes=32 time=2021ms TTL=64
Reply from 192.168.0.1: bytes=32 time=5ms TTL=64
Reply from 192.168.0.1: bytes=32 time=7ms TTL=64
Reply from 192.168.0.1: bytes=32 time=1982ms TTL=64
Reply from 192.168.0.1: bytes=32 time=6ms TTL=64
General Internet browsing is “ok”, but can be slow.
Social media apps such as Facebook and Twitter will regularly lose connection.
Streaming video is a nightmare.
I’m about to move house and there is no way I will be using Virgin again.
I know technically its not their router, but the lack of communication from them is a joke.
Hmmmm, just rejoined VM after 3 years (got a great deal). Been with Sky and currently BT. Not a heavy user but multiple devices used with WiFi. Last time used SuperHub2 without issue. Hoping no issues with the SB3 although this thread has raised my concerns.
Good luck. I’ve been with the Hub 3.0 for 5 days now and it’s dreadful. I’ll be switching back to my hub 2.0 ASAP.
Hi all. I recently upgraded from hub 2.0 to 3.0. This was because of my hub 2.0 constantly needing to be rebooted. Wireless signal is now much weaker, speeds upstairs much slower. General freezes all around, Some devices unable to connect at all and frequent resets required. I haven’t even been in touch with VM yet.
My idea is to switch back to my hub 2.0 and put it into modem only mode, then buy a separate router (eg. Netgear Nighthawk).
Anyone think this is a bad idea?
My SH2 has failed (might be power plug, might not be) and VM have only offered me a V3 hub, no option of a replacement power adapter. Bearing in mind the obvious issues with the V3 hub, where can I get a replacement power adapter from? Initial searches haven’t shown any options for this
Household with 4 active gamer here. We were on the 200M connection with a SH2AC, everything was just lovely SH2AC -> Draytek 2230 -> gigaswitch -> PC’s and consoles.
We upgraded on saturday to 350M with a SH3, worst error I’ve made in a large while, I had failed to research the SH3 :/. Large latency spikes, dropped packets and many many games ruined by the SH3 deciding to reboot every now and then.
Did some research (wrong way round I know).
Friday contacted VM asked to cancel my upgrade, and be switched back to 200M using my SH2AC, after quiet a bit of time with them denying that it would be possible to reactivate my SH2AC they finally relented and admitted that yes it might be possible but ‘more than my jobs worth’ after a bit of polite badgering they did reactive it.
Now everything is back to normal, if you game at all do your best to avoid the SH3, hopefully before my SH2AC breaks down they’ll have a decent SH4 available.