One faulty Openreach ONT left several UK homes without broadband for a month
Did you know that a single rogue Optical Network Terminal (ONT) inside somebody’s home can sometimes cause a more widespread broadband outage? Sadly, this is exactly the sort of problem that one of Sky Broadband’s subscribers recently faced after they experienced a month long outage on Openreach’s full fibre (FTTP) network. But they weren’t the only ones impacted.
The ONT (or ONU) device is an optical modem that’s usually installed inside your home or office, near to where the fibre optic cable physically enters your property. The primary job of this kit is simply to take the optical signal and convert it into an electrical one that can then be connected to your broadband router via an Ethernet (LAN/WAN) port. The standard ONT is usually a very small single port device for one FTTP line (pictured).
Now a lot of things can go wrong that might disrupt an ONT’s ability to function correctly, such as a power cut; your network operator botching a firmware update to the device; or deploying the wrong ONT (mixing up houses in the database) etc. But those aren’t the only potential faults and one of the rarest can result on a rogue ONT that harms the connectivity of more than just a single home.
Advertisement
In this case the problem was first raised to ISPreview by one of Sky Broadband’s customers, Matt in Thornbury (North Bristol), who complained that his Openreach based full fibre broadband service had been offline for a month and he was struggling to get any useful information out of Sky. Apparently Sky merely pointed to the cause as being an unspecified fault within the local Almonsbury exchange and couldn’t say when it would be resolved.
Suffice to say that it’s unusual, particularly for somebody living in a sizeable market town like Thornbury, to be offline for that length of time due to an exchange fault. Such issues would normally be resolved within hours or days, not weeks or months. Equally confusing was the fact that we couldn’t spot any signs of major infrastructure damage that might have caused such a lengthy outage.
Enter the rogue ONT
The good news is Openreach were significantly more helpful than Sky and revealed how, after an investigation, they had discovered that the problem was caused by a single faulty ONT, which was causing disruption to multiple services on the area’s shared FTTP broadband network (four customers to be precise). The exact nature of the fault wasn’t explained, but from the description we can surmise the cause.
In a healthy fibre network the main exchange transmitter (OLT – Optical Line Terminals) controls traffic by assigning precise time slots to each house’s ONT. Your ONT is normally only allowed to “talk” and transmit light upstream during this micro-window. But sometimes a fault can occur that causes an ONT to constantly fire light back down the cable, which disrupts local connectivity (i.e. blinding the splitter link and exchange connection).
Advertisement
Such faults are rare and normally get resolved fairly quickly, partly because those with such a connection will usually report the problem. In this case the person with a faulty ONT, which was disrupting all of the local homes (including Matt’s), never actually reported that their connection was down (despite knowing it hadn’t worked since the day of installation).
Instead Openreach had to spend more time doing detective work to identify precisely where the problem was coming from and then isolate the related line. Thankfully they were able to do this and Matt was finally reconnected on Thursday last week. The operator also dispatched an engineer at the same time to replace the faulty ONT that caused all of the trouble.
At the end of the day, we knew problems like this could occur, but they’re hard to spot and this is the first such incident that we’ve been able to confirm. In theory modern network monitoring solutions should make it easier to identify issues like this, but the disruption can make that exercise more difficult (it still helps when customers report if their connections aren’t working). Quite why the user with a fault ONT in this case didn’t report it remains unclear.
Mark is a professional technology writer, IT consultant and computer engineer from Dorset (England), he also founded ISPreview in 1999 and enjoys analysing the latest telecoms and broadband developments. Find me on X (Twitter), Mastodon, Facebook, BlueSky, Threads.net and Linkedin.
« Fastr Broadband quietly launch 8Gbps full fibre package in City of York UK
Advertisement
Leave a Reply
Privacy Notice: Please note that news comments are anonymous, which means that we do NOT require you to enter any real personal details to post a message and display names can be almost anything you like (provided they do not contain offensive language or impersonate a real person's legal name). By clicking to submit a post you agree to storing your entries for comment content, display name, IP and email in our database, for as long as the post remains live.
Only the submitted name and comment will be displayed in public, while the rest will be kept private (we will never share this outside of ISPreview, regardless of whether the data is real or fake). This comment system uses submitted IP, email and website address data to spot abuse and spammers. All data is transferred via an encrypted (https secure) session.
You’d think this is a situation that would be automatically flagged up by network monitoring, if it isn’t then someone could maliciously replace their ONT with a device that deliberately behaves like this faulty one and take down the connections of everyone on the same OLT.
In a GPON network it’s not always that easy. Sometimes the customer with the faulty ONT may actually still have a working connection, so you can’t assume it won’t be operational. In other cases that same ONT may only be intermittently faulty. We don’t know the exact specifics of this case.
The other issue is that the chaotic data collisions created by this can essentially end up masking the real source, which might require a process of elimination in order to track it down across the full optical splitter tree (i.e. briefly disconnecting individual lines). But that would be a last resort. Suffice to say that in the real world it can be much more tedious to track down.
I would have thought there would be some system to report the issue. The person who did not report the issue may have been away from home for a month or so,
so much for the more advanced full fibre networks.
Exactly or that the ONT had on-board software that would detect the “Out-of-Envelope” condition, whatever the cause i.e. component failure or ill-advised user twiddling and turn the unit off after reporting the condition to OR/Sky. So much for the resilence of the cheap, cheaper, cheapest data infrastructure.
That said, the 5G/4G wireless has similar problems for users, where, in urban areas, the plethora of cell tower masts means that people in fringe areas suffer highly variable speeds due to the 5G/4G constantly switching lock on towers due to overlapping coverage, marginal signal strength (S/N ratio 7-8 !)and over-subscribed cell usuage at certain times of day. In my location, the speed can vary on 5G NSA + 4G LTE between 250 mpbs and 600 mbps.And the only way of mitigating this effect is to be highly selective in the bands chosen . . and that’s only possible if you can isolate the towers being used – Cell Mapper and similar products are of little use in this situation. Current experience leads one to conclude that it would have been so much better if HMG schema for letting wireless coverage had been on the basis a competitive tenders for exclusive single operator operation per neighbourhood with a contract stipulation as regards universal service obligation within that neighbourhood.Or alternatively, perhaps, a wireless version of railtrack/openreach operating in the same manner as the various collaborative commercial associations in computing electronics comnponents industry do ?
And this is why network operators are generally very reluctant to go wires only (and especially not with untested, unapproved BYOD equipment as some on here wish would happen).
I am surprised that Openreach didn’t have a process in place to tackle it more quickly without customers needing to report the fault. I would assume their monitoring should be able to pick up if a group of ONTs (perhaps an entire PON) all go out of contact at the same time, as that’d signify something that needs immediate attention. Presumably the ONT’s manufacturer will be grilled over it too – as they’re supposed to have protections to prevent the laser staying on
When home broadband customers ask for wires only, they’re not usually concerned about the bridge (ONT), they’re more concerned about their router. I’m sure there are exceptions, but not at a high scale.
Also, as an example, AFAIK, in the US where consumers are allowed to bring their own terminals, the provider is allowed to pre-qualify compatible hardware that customers can pick from when shopping.
As apparently only 4 customers had been affected it might be reasonable to assume there were only 4 connections to that particular OLT terminal. If it had been the full 30 connections and 30 had been taken offline at once maybe it would have raised a bigger flag.
Those US ISPs you refer to are generally cable ISPs. The US cable companies permit (are required to?) use of a customer owned cable *modem* or integrated modem/router so that customers can save money on rental costs.
Cable networks are a shared medium so, just as with PON, they choose to be careful as to what they let onto the network. The cable co retains full control of the customer owned modem – eg the customer can’t manually update the modem firmware.
UK ISPs don’t charge for equipment rental so this is not a concern, and “wires only” in the UK context literally means that. You have to supply your own ADSL/VDSL/G.fast modem and Openreach requires ISPs and manufacturers to submit VDSL or G.fast equipment for testing.
There may be a possibility that Openreach eventually allows ISPs to do the same with FTTP with routers that have an integrated ONT, but that’s some way away yet.
Customer devices are a red herring. Equipment can go wrong no matter who provides it. In fact statistically, it is far more likely that this is a fault with the OR provided equipment. With millions of ONTs out there, some will fail and some of those will fail in a way that causes an issue. OR need to be able to deal with any sort of failure.
The real question is why OR took so long to sort it out with just four affected properties. Assuming that the OLT diagnostics didn’t indicate the rogue ONT and remote commands weren’t working, they could easily go out and just replace all four ONTs completely.
It feels like this is a case of OR spending as little money and effort as possible to rectify the situation.
“I am surprised that Openreach didn’t have a process in place to tackle it more quickly without customers needing to report the fault.”
Careful, BT Ivor, you may have to be re-programmed 🙂
I agree with what you said though, you’d think some kind of network monitoring would pick up on this, OR the software in the ONT would have a watchdog service if it detected light saturation through continuous transmit over a long period. Not sure of the exact detection method.
Ben – should be near enough 100% probable to be Openreach equipment since they currently do not allow any other ONTs on their network. (yes, some people have figured out ways to do things they shouldn’t, but still)
The point I am making is that currently OR has a direct relationship with their ONT manufacturers and can get answers from them as to why this occurred, whether it might happen again, and thus whether any action needs to be taken eg to organise proactive ONT swap outs. With wires only they’ll have none of that, especially not from people who want to plug in the finest no name XGS/GPON SFPs.
I’ve not seen anyone talk about using their own ONTs outside of situations where the provider doesn’t provide a ‘modem mode’ or a separate modem/ONT e.g. Virgin Media on FTTP or cable companies abroad.
If the companies provide these then I doubt it will be much of an issue outside of some fringe cases.
“Your ONT is normally only allowed to “talk” and transmit light upstream during this micro-window.”
Out of interest, what is the duration of this “micro-window”, and how often do they occur?
Also, do Alt-Nets such as City Fibre use the same Time Division Multiplexing principle?
According to this cisco doc on GPON, 125u seconds
https://www.cisco.com/c/en/us/support/docs/switches/catalyst-pon-series/216230-understand-gpon-technology.html
>> Also, do Alt-Nets such as City Fibre use the same Time Division Multiplexing principle?
Yes as it’s part of the GPON technology that they all use. But any kind of shared transmission medium can be jammed by a broken (or indeed rogue!) transmitter no matter what multiplexing technology is used.
@john_r – CityFibre stopped installing GPON to new customers over a year ago now. They use XGS-PON, but back to the original question that also uses Time Division Multiplexing or more accurately Time Division Multiple Access on the upstream. 50G PON also uses the same approach.
This might get interesting for somebody technical and doesn’t like their neighbour. They could screw up the entire PON every time they go to bed / on holiday!
A potential DoS attack vector you mean….
Knowing my luck if I did it I would affect the wrong next door neighbour. Not that I can because i am on a different network.
This could get fun on XGS-PON where the split ratios can be I think 128:1 rather than 32:1, finding the rogue ONT could be difficult, depending on how they do the splitting(if they have a 1:4 and 4x 32 way splitters, they could narrow down to 32 customers)
I wonder if we could also see essentially vandals who remove an ONT and replace with something with the laser always on
Isn’t 64:1 possible on GPON already?
The actual split size is upto the operator however
Faulty ONTs and other failure modes are a known factor. A good ONT will have self-diagnostics so that it can see if there is an issue and shut itself down. On top of that, depending on the manufacturer the OLT (in the exchange) has quite a lot of information available. For example it can detect if an ONT is transmitting outside the window it has been given or otherwise ignoring instructions. Then it can be rebooted, or even completely disabled remotely to see if that solves the problem.
The only real issue is the unlikely case where an ONT locks up and doesn’t respond. But normally by then a customer is on the phone and someone in customer services is telling them to restart it. Or arranging an engineer visit.
The tech (both GPON and XGS-PON) allow 1:128 maximum splits. Openreach’s policy is max 1:32. Altnets are anyone’s guess.
Why 1:32? Well, But 2488mbps GPON divided by 32 is 77.75mbps guaranteed share of bandwidth. Ever seen 80mbps down before on OR services?
presumably the only way to isolate the related line is to physically remove it?
I have a 2.5gbps switch which locks up my entire LAN and internet access when a slower speed like 10mbps is negotiated on any of it’s ports.
Faulty / bad devices are a real disruption!
Props to Openreach for working it out here.
“Quite why the user with a fault ONT in this case didn’t report it remains unclear.”
Our business signed up for service with Fibrus (the only fibre provider in our area) at 2 separate sites. Unfortunately, the service has constant low level packet loss (around 0.5 to 1%), severely slowing down transfers. Perhaps it’s how their bandwidth management works, who knows – no one at Fibrus seems remotely interested in fixing it.
Anyway, we managed to cancel one of the sites, but missed the cancellation window on the other by 1 day.
So occasionally (maybe every couple of months) I fire up a PPPoE connection over the ONT, run a few tests with curl and iperf, and marvel at downloads which continue to be faster over 800 MHz LTE from a transmitter 5 miles away than via our “900 Mbps” Fibrus service.
The ONT is powered up, and connected, all of the time. It could “go rogue” at any time, and I wouldn’t notice. I’m not sure Fibrus would either.
BYOD? Surely that refers to providing your own Router. The service I purchased was exactly that. Providing your own ONT seems slightly different in scope.
My internet keeps going off
Keep it away from naked lights, it’s nearly halloween. 😉
Only a month ! Been without broadband for 7 weeks and counting after open reach installed my full fibre broadband to another
Property 50m away.
Sky and openreach still not have me up and running ONT box and fibre fitted after 5 weeks still as of today not working, nothing but excuses from both sky and openreach.
A fault or issue in one household that impacts other users……..Anyone remember when the other household in the shared party line had left their phone off the hook? You couldn’t make calls as you couldn’t lock the line, and you couldn’t phone the operator as the you needed a line to do so. It was OK if they were in, and you could run round and knock on their door. However, if they left the dog in and went out and the dog knocked the reciever you were stuck…..
I remember those days, we never had had a party line, but I remember in the back of my mind someone who did, I think it was my Mums friend, but I could be wrong. Also, could pick up the phone and hear someone else talking on the other side 🙂
“Apparently Sky merely pointed to the cause as being an unspecified fault within the local Almonsbury exchange and couldn’t say when it would be resolved.”
“The good news is Openreach were significantly more helpful than Sky and revealed how, after an investigation, they had discovered that the problem was caused by a single faulty ONT, which was causing disruption to multiple services on the area’s shared FTTP broadband network (four customers to be precise). ”
What was sky meant to tell the customer when it actually turned out to be an unspecified fault within the local shared FTTP network?
wasn’t the sky explanation passingly accurate?
this is the issue when riding on someone else last mile connectivity. It becomes pass the buck time and the ISP is beholden on the actual last mile provider to do the investigation.
I used to hate using 3rd party providers for enterprise WAN’s, with BT IT was wholly on them to provide answers, with Verizon/C&W/Vodafone etc they always blamed delays on BT mainly because it was BT doing that last mile. With BT often the 2/3 line support had access to the full stack in the path and I could get useful info instead of with the others BT would update & then I’d have to wait for that 3rd party to update me. It was a while back & now that BT & Openreach are separate entities I have no doubt it’s the same delay process with BT.
I hadn’t appreciated that GPON was TDM from the ONT, I had assumed it was only aggregated in a street cabinet & home to the cabinet was dedicated.
it makes some sense aggregating by TDM but does pose some risks like this tale relates.
As others have mentioned you’d expect the faulty ONT to fail safe.
Some may have multiple lines for redundancy and may have been happily working on a primary link or maybe one of those routers with 4/5G backup & hadn’t noted they where using cellular data because the fibre was failing. Or they where on holiday, who knows.
Something new learnt.
the fact a failing ONT can take out multiple customers and the ISP has no clue as can only see their router while OpenReach also have no clue as they don’t get the errors from their ONT.
Yeah I did think the article was a bit harsh on Sky. They can only relay what OR tell them. What I’d like to know is did Sky payout the circa £300 automatic compo due for a months outage?
This has happened before, although it is rare.
I dealt with an issue like this a few years ago that had intermittent drops and lock up’s overnight that we traced to a rogue ONT.
New build estate, seven customers reported having issues. One of the home owners had accidentally smashed the ONT when moving in and bought an old multiport ECI model from eBay and plugged it in. Ironically they had a go live date planned for about a week later.
The reason this one was so hard to find was it had 28 ONT’s connected but only 13 were live with a working service, and not all of them realised they had an issue.