Network access provider Openreach (BT) has just informed us that more broadband ISP traffic was carried across their UK network yesterday (Sunday 11th December 2022) than ever before – totalling 229PB (PetaBytes), which compares with 222PB on the previous highest day (5th December 2021).
Openreach typically supplies numerous broadband ISPs, mobile network operators and other businesses across the country. As a result of all that, their network often sees the impact of big events, as well as any key changes in consumer or business behaviour. “It’s thought people getting into the festive spirit is behind the peak, as they headed online to shop, stream Christmas music and download festive films,” said a spokesperson for the operator.
Sadly, the operator doesn’t say anything more than that, although it’s worth noting that Sunday also saw much of the UK being blanketed by snow and keeping people at home (this tends to turn internet connectivity into the primary entertainment medium, except for snowball fights etc.). We should point out that demand for data is constantly rising, and so new peaks of usage are being set all the time by network operators (usage typically grows by 30%+ each year).
Advertisement
Part of the problem is Terrestrial and Satellite broadcasting are moving online, that’s in addition to the streaming services like Netflix, Disney, Prime, Youtube and others.
My guess is, providers offering Gigabit services are banking on customers utilising network at different times, and are hoping their backbone and router networks will be able to cope with average levels of traffic 99.9% of the time, but live online broadcasting would contradict that view, especially for live sporting events. Imagine multiple simultaneous sporting events spread across different channels, like in the olympics, with many different household viewers watching different events in different rooms, all in high definition up and down the country, that would test even the best networks.
The BBC are now talking about moving to an online only service. Sky are heavily pushing Sky Glass, will they eventually drop Satellite transmission in favour of a full online service? The U.S. based subscription streaming service providers are offering ever higher definition streaming services, so all of this will put further strain on all Telecom provider networks.
OFCOM will need to pay close attention to this, since the Net Neutrality legislation wasn’t designed around supporting multiple broadcasters, offering multiple channels, utilising ever higher bandwidth over the internet. It’ll be interesting to see how the debate unfolds going forward, will internet traffic be classified into types with broadcasters expected to buy extra leased IP/backbone capacity, to ease the bandwidth burden on the Telecom providers, or will the Telecom providers be expected to absorb the extra cost and pass it on to their customers through higher bills?
Currently the Telecom providers are being lambasted for the CPI+3.9% hike, but their backbone and cloud operating costs increase in line with bandwidth, who pays for that and shouldn’t that be taken into account when the media criticises increases in broadband bills?
Bandwidth use has steadily increased since the availability of the internet and bandwidth cost correspondingly decreased. Much of the bandwidth demand can be cached on an ISPs network without incurring external transit, video on demand and apps/games in particular.
If the BBC moves to online only, the concept of a broadcast becomes irrelevant. You don’t need to have millions of streams traversing the internet, you serve the video from a local cache. It’ll also be a gradual trend, as it is already with more viewing moving to iPlayer and other IP platforms. There isn’t going to be a one day cutover. Net neutrality has served its purpose fine up until now despite naysayers, there’s no reason to think anything different is needed.
In contract price rises move all of the risk onto the end user. I’m locked into a service for however long without knowing how much I’m going to be paying for it in a years time. CPI could be 5%, it could be 20%. It also guarantees a price rise mid contact, as conveniently a fall in CPI would never be passed on. The price increase is rarely reflected in new customer pricing either, so the rising costs can hardly be that biting. CPI is also a broad measurement rate, not necessarily reflective of the the providers increase in costs. The increase in connection cost then feeds back into inflation, increasing the upward pressure on it.
If you want to increase prices, don’t lock people into a contract. If you can’t provide the bandwidth to back the connection speed you’re selling, don’t sell it. It’s really just a basic principle of fairness.
I don’t understand what their problem is, it’s not like the ISP or openreach created the content or has to encode the video in real-time requiring all that power… all they’re doing is moving packets from one location to another… it’s like setting up a small LAN network in your home, use 2 ethernet cables and transfer 1tb file or files, the 2 computers on both ends are doing more work than that little 8 port switch in the middle… obviously their network is much more complex. It seems an easy enough thing to deal with.
I think they should upgrade their network specially the switches so there is much more capacity available before throttling every connection, and identify all the bottlenecks and ancient hardware from WW2… simple really.
“Much of the bandwidth demand can be cached on an ISPs network without incurring external transit, video on demand and apps/games in particular.”
That’s fair enough, but who should pay for the caching servers? Should the “broadcasters” get together to pay for the servers and maintenance? Should the Telecom providers install caching servers and charge streaming content providers for use, to guarantee Quality of Service levels? Should the Telecom providers install, power, and maintain the servers, spreading the cost around all their customers including the ones that don’t stream content? Which would be very few if “broadcasting” moves completely online, which is where things appear to be heading.
“ all they’re doing is moving packets from one location to another”
Have you ever heard of the Dunning Kruger effect?
‘That’s fair enough, but who should pay for the caching servers? Should the “broadcasters” get together to pay for the servers and maintenance?’
That’s what happens with Netflix, YouTube and others right now.
https://openconnect.netflix.com/en_gb/ is an example.
“That’s what happens with Netflix, YouTube and others right now.”
OK, so Netflix have installed four Caching servers in London data centers, in the case of the UK, but there are no details about backhaul connecting to local Geographic areas. Are the Data centers backhauling the Data to Birmingham, Liverpool, Manchester, Edinburgh, etc? Or is the content streamed out in London and left to the carriers to deal with the data? I see they say they have agreements with ISP’s, but it’s short on detail. The caching servers in London are likely aimed at reducing traffic on International subsea routes, rather than easing congestion on the national networks.
You missed this bit:
‘Embedded Open Connect Appliances (OCAs)
Open Connect Appliances can be embedded in your ISP network. Embedded OCAs have the same capabilities as the OCAs that we use in our 60+ global data centres, and they are provided to qualifying ISP partners at no charge.’
‘OK, so Netflix have installed four Caching servers in London data centers, in the case of the UK’
No detail on where the caches live though the private peering page may be a hint. The answers to the rest are at https://openconnect.netflix.com/en_gb/peering/#locations
John…
‘I think they should upgrade their network specially the switches so there is much more capacity available before throttling every connection, and identify all the bottlenecks and ancient hardware from WW2… simple really.’
How much of that ‘ancient hardware from WW2′ besides copper cabling do you think is carrying Internet traffic?
Openreach aren’t going to throttle anything: they make their money selling bandwidth.
There are no bottlenecks on the Openreach side as far as switches go. The only restriction is the copper between cabinets or exchanges and homes.
Any issues with congestion are once the data leaves Openreach control and is on their customers’ networks. Those are the networks that have to send data around the country, not Openreach.
“You missed this bit:”
Yeah I missed that, it appears Netflix are doing some of the right things, hopefully others will do similar. That might alleviate bandwidth issues on the backbone Networks, but I still foresee issues for the local core routers connecting to the increasing number of caching servers, and access routers distributing ever increasing streaming media content received from the core.
Netflix say “We provide the server hardware and the ISPs provide power, space, and connectivity”, but it’s debatable that ISP’S should have to pay, house and connect servers for the benefit of the content providers. The ISP routers and switches utilise more power as the bandwidth ramps up, so siting servers at ISP’s is more for the benefit of the streaming content providers than the Telecom companies, as the content providers are protecting their Quality of Service model, but passing the server running costs onto the ISP’s. Shouldn’t Netflix pay the ISP’s to house, power and connect their OCA’s?
This is a complicated state of affairs and we can only guess at the issues, but judging by various articles the Telecom companies and ISP’s don’t appear to be happy with the current situation, hence complaints to the various regulators around Net Neutrality. I suppose opinions will differ, dependent on personal vested interest.
‘That might alleviate bandwidth issues on the backbone Networks, but I still foresee issues for the local core routers connecting to the increasing number of caching servers, and access routers distributing ever increasing streaming media content received from the core.’
Then you do what Sky have been doing and move the content delivery network closer to the customer: they have some exchanges with caches in them.
‘…it’s debatable that ISP’S should have to pay, house and connect servers for the benefit of the content providers.
The ISP routers and switches utilise more power as the bandwidth ramps up, so siting servers at ISP’s is more for the benefit of the streaming content providers than the Telecom companies, as the content providers are protecting their Quality of Service model, but passing the server running costs onto the ISP’s. Shouldn’t Netflix pay the ISP’s to house, power and connect their OCA’s?’
They don’t have to pay: they can choose to have the traffic traverse their core, peering and transit which is probably going to cost considerably more than the power, space and cooling for caches. They could even intentionally allow the connectivity to Netflix to congest by not peering with Netflix at all but instead only accepting the route across a transit link with limited bandwidth.
The choice for ISPs in the UK at least is simple: our market is competitive. We want the content that Netflix et al provide. Netflix want to provide that content and are generally very happy to facilitate it with ISPs. ISPs can either get on board and be the conduit between their customers and Netflix or they can provide it at a lower quality in which case they’ll lose customers.
We pay our ISPs to connect us to content providers. Without the content there’s no Internet and, hence, no revenue stream.
In the USA ISPs were able to get away with this double dipping as they had very little competition in many areas while in the UK and Europe this just isn’t the case. I’m bemused as to why BT, and I believe they’re if not the only voice certainly the most prominent source of complaints in the UK, are complaining. BT Retail pay BT Wholesale for bandwidth. BT Wholesale charge for ports and per Mbps per month. Openreach charge BT Wholesale for connection to the exchanges and to the access network. There are CDN caches and extensive public and private peering. BT Group is paying itself for the bandwidth customers consume. It’s quite bizarre that the loudest voice complaining about Netflix specifically is the one with the most circular revenue stream.
On a more technical note BT Retail only have PPPoE to exchanges: if they were to modify how they hand out IP addresses, regionalise it, they could happily run with regional cache farms. Others using BT Wholesale have PPPoE to a few handover points so can’t readily reengineer their networks to break out into IP earlier so that they can host caches locally.
My main ISP are fine: they could stick a cache server next to the kit my FTTP terminates on, there’s a router in the exchange where my service breaks out into IP.
5 of 5 packets received 0 % packet loss Min: 1.338 ms Avg: 1.551 ms Max: 1.667 ms
My backup not so easily as they’ve NNIs in Manchester and London only so could only run cache farms there, however they’re also not remotely bothered by the situation.
@XGS Is On:
If the latency on your FTTP is 1.5ms over a couple of kilometers or so to the “local exchange” I’d be logging a fault….
The latency is where I would expect it to be given the equipment either side of the fibre so no fault to be raised.
If it were a microwave link with everything tuned for high frequency trading then I’d be concerned.
“I’m bemused as to why BT, and I believe they’re if not the only voice certainly the most prominent source of complaints in the UK, are complaining.”
It isn’t just BT complaining, the link below from September, which you commented on, says there are a lot more across Europe. In the UK Vodafone and Three are also a prominent voices, although Three isn’t mentioned in the first link, but is in second link.
https://www.ispreview.co.uk/index.php/2022/09/bt-and-vodafone-uk-reiterate-desire-to-make-netflix-etc-pay.html
Here’s another article from January talking about OFCOM’s review. Mobile operators are particularly affected by streaming content utilising excessive bandwidth.
https://www.ispreview.co.uk/index.php/2022/01/isps-and-content-providers-respond-to-ofcoms-uk-net-neutrality-review.html/2
Mobile operators have slightly different rules on network management and, of course, no-one put a gun to Three’s head and forced them to sell unlimited data for 50p a day.
Three’s search engine listing starts ‘No speed limits. Just endless streaming, browsing, and sharing in the UK.
Ya.
There are a few big telecoms operators in Europe who want to decide what you can and can’t use your internet connection for, by sitting in the middle taking money from both the customer and the content provider.
There is only a debate in Europe at the moment because big Telco CEOs have been cosying up to EC Commissioner Thierry Breton, ex-CEO of France Telecom, who seems to see this as a way of extracting money from US tech companies.
Net neutrality advocates think it’s a terrible idea, consumers associations think it’s a terrible idea, small telcos think it’s a terrible idea. The only people who want it are the big incumbent telecoms operators.
Oh, there’s also an Ofcom Net Neutrality consultation on at the moment, where you can tell Ofcom it’s a bad idea too.
The problem of CPI+3.9% hikes is not the hike itself but the totally ridiculous idea that you can contract a 12/18/24 months service and the service provider is unable to estimate the cost of proving such service at a fixed price for the duration of the contract. The price increases should also be in line with the increased cost of proving the service not some random fixed formula. Finally I see that existing customers get ripped off with price increases while new customers get the same old prices or even lower. This system has been developed to rip off customers who do not check their bills, sign new contracts or look for better prices. In other words the illiterate, the lazy or the forgetful. That’s not a good way to treat your customers.
All TV will moving online soon. BBCi, ITVX, All4 and My5. The aerial and satellite will be switched off.
Sad to think only 60MB of it was mine!
Just the natural growth of the Internet and demand for data. It has been growing like this for decades and will continue to do so. As devices become more advanced, so does the size of the software payload to support the hardware. Downloads naturally become larger, video games become more complex, textures get larger. Streaming video advances, 4K.. 8K.. 16K.. Ad infinitum.
Think the point is more that there’ll be a spike when we go IP-everything. If the migration is relatively gradual it won’t be too bad: when broadcast goes off it’ll be clearing up a relatively few customers remaining.
See https://networkingchannel.eu/living-on-the-edge-for-a-quarter-century-an-akamai-retrospective-downloads/ for a very good summary of the cacheing discussion (by the man who invented the concept).