Which? Claim UK ISPs Could Overestimate Broadband Speeds by up to 62%
A new study from consumer magazine Which? has used data taken from over 700,000 speed tests conducted by Speedchecker Ltd (between January and March 2017) to claim that broadband ISPs could be overestimating their service speeds by up to 62%. Save for a few BIG caveats.
The research, which also used data collected by Ofcom for its 2016 Connected Nations report (perhaps they should have waited for the imminent 2017 update), looked at the median download speeds in each local authority area, taken from the speed tests, and compared them with the median speeds that providers say people in the same areas can get, as reported by Ofcom.
In 52% of local authority areas, people using the speed checker experienced median speeds that were at least 10% slower than the median speeds estimated by ISPs in the report. Over a third of local authority areas (35%) had speed checker medians that were at least 20% slower than estimates. So where does their headline “up to” 62% figure come from? Apparently the gap was worst in Ashfield (Nottinghamshire).
Meanwhile, people using the speed checker experienced median speeds at least 10% higher than those given by providers in 16% of local authority areas. In the remaining 32% of local authority areas, the median speeds experienced by speed checker users were less than 10% different from the median speeds specified in the report.
Alex Neill, Which? Managing Director of Home Services, said:
“Our research has shown that in some areas there can be a big gap between what people may expect versus what they actually experience in their homes.
This gap raises questions on whether people across the country are really getting the service they are paying for.
People who are unhappy with their broadband should check their speed and follow our tips on how to improve their connection at home or switch to a better service.”
Which? correctly notes that consumers usually receive slightly slower speeds than those given by providers due to factors such as line length, interference and network performance, although they should really have gone a bit further to clarify some of the context and other key limitations with this data.
In particular many people will test their speeds over WiFi rather than wired connections and wireless networks tend to produce much slower results due to local signal disruption (walls, doors, distance etc.), which means that you’re not directly testing the broadband connection itself. Ideally the best way for end-users to test would be to do so at the router level, which removes a lot of potentially disruptive factors.
Lest we forget that poor quality home copper wiring is another factor that can reduce broadband speeds, albeit particularly on the older style of ADSL and FTTC (VDSL2 / G.fast) technologies. A lot of people still don’t properly filter out interference from the old bell wire and that can have a big impact on the speed you receive, although it isn’t the ISPs fault.
The speed test itself can also suffer instability due to issues with server performance or network routing / peering, which can impact the result. Not to mention that if somebody is conducting a test while others are using their home network then that will also result in lower speeds.
We’re also unclear about the context when Which? says they compared the speed test data with Ofcom’s information on the “median speeds that providers say people in the same areas can get.” We don’t know exactly what the source of that is and if they’re using the performance based off general network availability by technology, headline advertised speeds, sync speeds or the personal user speed estimates given by ISPs.
The best comparison may be to use the personal estimates provided by ISPs because these are specific to the end-user, although accurately collating personal estimates with the data gathered from public speed tests is currently very difficult to achieve without consumers agreeing to share such data with the testing websites. Likewise not all ISPs provide a personal estimate.
We can’t stress enough how important it is to get the context of such testing correct, otherwise the results may not say what Which? think. Suffice to say that we must always take such data with a pinch of salt and it would be more reliable if they could exclude tests conducted over WiFi (very difficult to do because speed testers can’t easily identify this without more user input).
A useful approach might be for speed testing sites to give users the option of offering some extra data, which would enable them to specify if the test was conducted over wifi, how many users are on their network at the time of the test, what kind of connection technology they have, what package they chose and what personal speed estimate had been provided by the ISP. Granted many would not provide this but if some did then the data would have much more value.
Mark is a professional technology writer, IT consultant and computer engineer from Dorset (England), he also founded ISPreview in 1999 and enjoys analysing the latest telecoms and broadband developments. Find me on Twitter, Google+, Facebook and Linkedin.
« ISP TalkTalk See Growth in Broadband Users to 3.99 Million and FTTH Plan
Comments are closed.
Privacy Notice: Please note that news comments are anonymous, which means that we do NOT require you to enter any real personal details to post a message. By clicking to submit a post you agree to storing your comment content, display name, IP, email and / or website details in our database, for as long as the post remains live.
Only the submitted name and comment will be displayed in public, while the rest will be kept private (we will never share this outside of ISPreview, regardless of whether the data is real or fake). This comment system uses submitted IP, email and website address data to spot abuse and spammers. All data is transferred via an encrypted (https secure) session.
NOTE 1: Sometimes your comment might not appear immediately due to site cache (this is cleared every few hours) or it may be caught by automated moderation / anti-spam.
NOTE 2: Comments that break our rules, spam, troll or post via known fake IP/proxy servers may be blocked or removed.
Sort of implies there is a link with the local authority and the information given.
Another dodgy survey.
Yet another problematic report on a tech subject from Which? It does seem to be hopeless when it comes to writing about this stuff, really needs to get some people involved that know what they’re talking about.
Move onto a fiber network – any network speed issues lies with the end user
Not really – after traffic leaves the exchange it’s still a ‘shared’ service and contention still exist in the backhaul, ISP network and peering.
If you’re on a 330Mb FTTP service, it doesn’t mean that you will be able to down a new iOS update at the headline speed.
problem sorted
On asking if people are using Wi-Fi, been there done that and the number using a mobile phone who insist its connected by Ethernet mean you cannot trust the data.
Just like asking the public their connection speed, so many don’t know you get random answers.
Good point Andrew.
Speedtest.net can only perform a speed test on I devices via the app, so they know if your on WiFi or the mobile phone network. Maybe OFCOM or which should invest in an app for pc’s that can be invoked from a web page and can automatically ascertain if the test is from WiFi or Ethernet.
That’ll be the only way of getting a more accurate result.
Of course the best way is to measure the speed from the users router, but until cheap domestic routers come with that capability we will have to make do with the inaccurate results that are then sensationalised by institutions pretending to be an sme in the technology.
A further point is that the Ofcom data usually does not discern between what technology offers and what person has bought e.g. in an area where VDSL2 can do 62 Mbps a user buying 38 Mbps should speed test in the 36 to 37 Mbps region.
I wonder if Which took this into account, plus also the even wider issue this creates in cable and full fibre areas.
Is it little wonder that Ofcom ignores web based speed testing if big names can make such a hash of a subject area.
How about some ‘thinking outside the box’ by the ISPs and Ofcom to resolve this repeating issue. We know these poor quality speed reports are going to continue unless something major changes. We know many consumers don’t know how to properly test their line speeds, and frankly they shouldn’t have to be techies to be able to confirm that the broadband speed sold by their ISP is the speed they are getting. So why not give the consumers the proper tools to test it with.
One possible solution:
1) Test line speeds in the opposite direction from internet to the router, the same way ThinkBroadband Quality Monitor ping test does.
2) The ISP’s router has a physical switch the consumer can operate to set the router to respond to ISPs speed tests.
3) The consumer plugs the router in to the Openreach master socket, if it isn’t connected already.
4) The ISPs obtain an Ofcom approved broadband speed testing tool and adds it to their website and then consumer logs in to the ISPs website and performs a speed test.
5) The only thing I can think of that is left that could impact performance is the house wiring, well we know the BT NTE master socket can be unscrewed and the router re-plugged in to the socket behind well how about Openreach making the master socket far more user friendly and change this to a simple physical switch on the side to disconnect the house wiring so the consumer can easily operate.
I know this isn’t a perfect solution but goes a long way forward in helping consumers give better data to Which and other customer focus groups, even if they only switch the route to speed test mode. This beating by Which and others is damaging to the ISP industry, making out ISPs are giving out fake performance figures which we know is not fair. As we know some of the readers here work for ISPs so I’m trying to get their attention and ISPReview’s website is a great area to bash out some fantastic ideas and solutions to solve some very difficult industry problems, and hence my ‘thinking outside the box’ suggested solution to resolve this on going industry problem with broadband speeds.
NOTE: thinkbroadband quality monitor is NOT testing line speeds, it is measuring latency, and that highlights the difficulty those of us in the testing industry have…i.e. poor understanding from many areas including Which!
Hi Andrew, first many thanks for your time to give me your helpful feedback. Yes of course ThinkBroadband does not measure speeds and it can’t without the router’s firmware modified to support this feature, and limited to only ISP’s IP addresses to stop DOS issues.
I do feel the frustration for both parties, ISPs and consumers. There needs to be a benchmark way of testing broadband speeds, both industry and consumers are wasting so much time on this issue. Think how much support time could be saved per ISP. Ofcom needs to ratify a benchmark maybe the ISPA could put forward an initial benchmark to Ofcom, currently as you say it is like trying to nail jelly to the wall. Any ISP’s out there that could press the ISPA on this matter, benchmarking line speeds?
Ofcom has a benchmark way that it already uses the SamKnows system, this is measuring throughput, which is of course different to line speeds. Though Ofcom reports do sometimes get a bit confusing in this aspect.
Hi Mark Jackson,
As I’m sure you want to help improve the ISP industry can you mention some of the discussion points raised here in the next damming consumer report for ISPs.
1) There is no industry standard line speed and quality benchmark tool for Which, other consumer groups and users to use, as a result they are coming up with inaccurate measurements.
2) The ISPA is there to promote ISPs so logically needs to take ownership of this issue and urgently find a benchmark solution to stop the bad user posts and unfounded consumer reports damaging the ISP industry.
Many thanks,
Alex