EE Tops Tutela’s Q3 2021 Benchmark of UK Mobile Networks UPDATE

Crowdsourced benchmarking firm Tutela has published their latest Q3 2021 study of mobile broadband (4G and 5G) performance across the United Kingdom, which found that EE “dominated the majority of the metrics tested,” but Three UK are challenging when looking at the latest ultrafast 5G services.

The latest report is based on data from the 1st March to 31st August 2021 period, which saw the firm take over 194 billion measurements, including 3.1 million broadband speed tests (conducted via third-party apps on Smartphones). Unlike previous studies, this one also includes the performance of 5G networks for the first time.

We must stress that any testing conducting via non-dedicated apps like this could be less accurate than dedicated ones (e.g. opensignal). Crowdsourced data can also be affected by the user’s location, as well as any limitations of the device being used, and it similarly lacks a common type of hardware (useful when trying to form a solid scientific baseline). Nevertheless, there’s still some useful data to be found in Tutela’s study.

However, rather than look at the usual categories today, we’re just going to skip right to the Key Performance Indicator (KPI) results, which provided most of the source data for the study and tell us much of what we need to know. Clearly EE come top for 4G performance and that tends to be reflected by other studies too, although it’s a different story for 5G where Three and O2 have the edge.

Overall KPI Results

5G Only KPI Results

Some readers will note that the median broadband speeds are significantly lower in Tutela’s study than in other benchmarks, particularly on 5G services. We think, where comparable medians exist (instead of means), this is more likely to be a reflection of their test configuration.

In addition, the KPI table above does not include all of the KPIs that are used in their “Consistent Quality” (CQ) calculation. For example, the Reliability metric is not shown in the KPI table. The CQ score for each operator reflects the percentage (%) of tests in common coverage areas that meet their CORE and EXCELLENT service thresholds – CORE reflects moderately-intensive tasks (i.e. web browsing, social media, SD video streaming), while EXCELLENT reflects more demanding use (i.e. HD video streaming, Skype video calls, online video gaming etc.).

CQ Results

Excellent
EE – 73%
Three UK – 67.8%
O2 66.2%
Vodafone 65%

Core
EE 89.5%
O2 85.8%
Three UK 85.3%
Vodafone 83.9%

UPDATE 4:11pm

According to Tutela, the reason why their download speeds are lower than other companies is due to their test configuration. “For our public reports, Tutela uses the results from a 2MB file download test. This is representative of typical traffic packet sizes that applications use. For example, Netflix chunks videos into files of around 2MB for buffering. Web pages are around 2MB in size on average… etc. Also, we run our tests against major content provider servers (e.g. Amazon, Google, Akamai) rather than a test server in an optimised location.”

The company claims that dedicated testing apps (e.g. Ookla Speedtest) “transfer a massive amount of data across multiple connections, to a test server in an optimised location (e.g. at the edge of the operator’s network) in order to measure the maximum potential speed of the network – but this does not represent the performance that a user would get with a typical application (except perhaps for downloading a very large file, which is a rare).”

Tutela further states that collecting data from software embedded in third-party apps is “not less accurate than collecting data from dedicated apps,” which is based on data tracked via their own dedicate app (here).