Ofcom UK Hit BT with £17.5m Fine for 999 Call Handling Fault UPDATE
The national regulator, Ofcom, has this morning hit BT with a fine of £17.5m for being “ill-prepared to respond to a catastrophic failure of its emergency call handling service last summer” (here), which readers may recall occurred after a technical fault (“complex software issue”) resulted in 14,000 emergency calls being disrupted. The disruption lasted for 10.5 hours.
Just to recap. BT previously revealed that a software bug had caused a “caching issue“, which resulted in impacted calls (i.e. those made to both 999 and 112) not being routed correctly and the user’s call being disconnected. At the time, a “robust temporary fix” was put in place to rectify this, and they’ve since implemented something more permanent.
The incident, the first in nearly 90 years, has already sparked the introduction of further safeguards to help “bolster the resilience” of the 999 emergency call handling system (here). But Ofcom has also continued to conduct its own investigation, which has now concluded.
Advertisement
According to the regulator, on Sunday 25th June 2023, BT experienced a network fault that affected its ability to connect calls to emergency services between 06:24 and 16:56. During the incident, nearly 14,000 call attempts – from 12,392 different callers – were unsuccessful.
The Three Stages of BT’s 999 Fault
Phase 1, from 06:24 to 07:33
During the first hour, BT’s emergency call handling system was disrupted by what was later found to be a configuration error in a file on its server. This resulted in call handling agents’ systems restarting as soon as a call was received; agents being logged out of the system; calls being disconnected or dropped upon transfer to the emergency authorities; and calls being put back in the queue. BT was initially unable to determine the cause of the issue and attempted to switch to its disaster recovery platform.Phase 2, from 07:33 to 08:50
The first attempt to switch to the disaster recovery platform was unsuccessful due to human error. This was a result of instructions being poorly documented, and the team being unfamiliar with the process. The incident grew from affecting some calls to a total outage of the system.Phase 3, from 08:50 to 16:56
The rate of unsuccessful calls decreased once traffic was migrated successfully to the disaster recovery platform. However, usual service was not fully restored initially as the disaster recovery platform struggled with demand.
Ofcom’s investigation found that BT “did not have sufficient warning systems in place” for such incidents, “nor did it have adequate procedures for promptly assessing the severity, impact and likely cause of any such incident or for identifying mitigating actions“. The regulator also found that BT’s “disaster recovery platform had insufficient capacity and functionality” to deal with a level of demand that might reasonably be expected.
The incident also caused disruption to text relay calls, which meant people with hearing and speech difficulties were unable to make any calls, including to friends, family, businesses and services. This left deaf and speech-impaired users at increased risk of harm.
Suzanne Cater, Ofcom’s Director of Enforcement, said:
“Being able to contact the emergency services can mean the difference between life and death, so in the event of any disruption to their networks, providers must be ready to respond quickly and effectively.
In this case, BT fell woefully short of its responsibilities and was ill-prepared to deal with such a large-scale outage, putting its customers at unacceptable risk.
Today’s fine sends a broader warning to all firms -– if you’re not properly prepared to deal with disruption to your networks, we’ll hold you to strict account on behalf of consumers.”
Despite the fact that no confirmed cases of “serious harm” have been identified as a result of the incident, Ofcom correctly makes clear that the potential degree of harm was still “extremely significant” and hence the large fine. The fine includes a 30% reduction as a result of BT’s admission of liability, full cooperation, willingness to engage, system improvements, regular updates and agreement to settle the case.
Advertisement
BT now has two months to pay the fine, which will then be passed on to HM Treasury. But the hope is that something like this will never happen again. We have contacted BT for a comment.
UPDATE 9:07am
We’ve now recieved BT’s comment.
A BT Group spokesperson told ISPreview:
“We take great pride in underpinning the national 999 service and recognise the critical importance our infrastructure plays. The level of disruption to the service on Sunday 25 June last year has never been seen before and we are sincerely sorry for the distress caused. We accept the specific points raised in Ofcom’s findings, and have put in place comprehensive measures to prevent this series of events reoccurring and improve end-to-end resilience of the system as a whole.
While no technology is 100% resilient, we have built a highly robust network with multiple layers of protection to connect the public to blue light services in their time of need. We take our responsibility to the emergency services and the public seriously, and on this occasion we fell short of our own high standards for the 999 service.”
Mark is a professional technology writer, IT consultant and computer engineer from Dorset (England), he also founded ISPreview in 1999 and enjoys analysing the latest telecoms and broadband developments. Find me on X (Twitter), Mastodon, Facebook, BlueSky, Threads.net and Linkedin.
« Preston City in Lancashire Establish Digital Infrastructure Cooperative
Brsk UK to Deploy FTTP Broadband to 30,000 Homes in Runcorn »
It’s fair enough that OFCOM should punish BT if their response failed in some way, but any additional backup/restoration procedure’s probably cost a lot more money. Between OFCOM nailing BT down and litigation like the CAT case currently awaiting judgement, it feels like BT are under attack from all sides.
It amazes me the amount of BT bashing that comes from all sides, and the number of people that join in wishing for the demise of one of the few remaining truly British companies. BT was probably one of the few UK companies that may have grown into a tech giant, but we as a country don’t appear to want national giants.
BT has a market cap around $18 Billion, Crowdstrike is currently valued at around $74 Billion; Crowdstrike’s revenue is/was expected to be around a fifth of BT’s, but the valuation given to the company is currently 5 times BT’s; This demonstrates why US companies take over the World, they can simply sell stock to finance acquisitions of more profitable, or upcoming, undervalued UK/foreign companies.
Since Crowdstrike set out disclaimers, in their terms and conditions, against any responsibility/claims for any losses incurred due to their software, affected customers have few options to recover losses due to the recent outage, so I would hope the World’s regulators dish out far harsher punishment than OFCOM regularly dish out to BT, since Crowdstrike have caused considerably more damage than BT could ever do.
British? Bull, may be based in the U.K, but look at the large shareholders and see how British they are. Is Patrick Drahi still one of the biggest shareholders of BT? He is certainly not British.
The only thing British about BT is British in their name, and they are registered here.
I don’t have any sympathy whatsoever for BT directors or the company itself. That fine is a pittance to them.
BT tried becoming a tech giant between 1999 and 2000. It didn’t end too well. They owed about £30 billion in 2001 and had to take various pretty extreme measures in order to remain solvent.
On the matter of comparing random US companies with BT if you compare more relevant companies, AT&T and Verizon, they are trading at price:earnings ratios of 9.94 and 14.57 respectively while BT are trading at 16.53. Much as BT were in 1999 when their shares hit over £10 each some other companies are potentially overvalued as they’re priced on the assumption of really rapid growth: that’s life.
Sorry if your stock in BT isn’t doing as well as you’d wish but point the blame somewhere else. BT are not excessively aggressively regulated relative to their European peers and are not victimised by anyone that could influence their stock price.
BT as one of the “few remaining British companies” – um, BT’s biggest shareholders are Patrick Drahi (French billionaire) at 24% and Deutsche Telekom at 12%. Mexican billionaire Carlos Slim also has 3%.
The reason BT has such a different market cap to Crowdstrike is it’s in a different market. Utilities that provide reliable low long term rates of return are valued less than high risk, high growth technology businesses like Crowdstrike (after last week, Crowdstrike is clearly high risk…).
One of the issues with BT is that they wish they were a technology company, rather than sticking to what they do best – providing connectivity. Hence Marc Allera’s excursions into trying to sell washing machines online and cloud services.
So, a once in 90 years event which was fixed in less than half a day results in a fine, which will end up being paid for by customers, yet no fines for the telecare companies for their continuing failure to adapt for the PSTN switch off.
Two-tier regulation by Ofcom!
Your name is apt because if you desperately needed a 999 service and it wasn’t there, the fact that they’ve provided good service for 90 years would be little comfort for you. With a mission critical system like that the recovery time should be minutes, not hours. The fact that there was no redundancy, no failover, no business contingency or continuity activating speaks volumes. Social media sites of police and ambulance services were being used to communicate the long dial number. BT have an entire mobile network they own. Why didn’t they use the emergency broadcast system to text out the long dial numbers based on area? Why did people have to access the internet to know about it?
because the network operators don’t have the authority to use that system (for anything other than test messages that 99.9% of users can’t see), and BT would have no means to send a message on the other three networks anyway
so, if this was a good idea – why didn’t the government or the emergency services themselves do it? They have their fingers on those buttons.
There’s also the question of whether these so called “long dial numbers” are a) something that the public should be aware of and b) have sufficient capacity. Where do these numbers route to? How do you ensure people in county or regional border areas get the correct number? How do you deal with instances where counties share one or more emergency services but not others etc. All worthy questions – but not in the heat of the moment.
And how much extra do you think the public would be prepared to pay for a 100% guaranteed emergency response system, even if that were possible? £500 a year? £5,000 a year? And don’t forget that the emergency services routinely fail to turn up after 999 calls, resulting in people dying. Should they be fined too?
I would rather the money went to organisations such as air ambulances; hospices etc rather than The Treasury.
“Fair cop we’re sorry” they have done a good job of keeping 999 served in the past. Dropped the ball on this occasion, maybe a bit of complacency? Shows the need for regular scenario training and testing if backup systems.
Guess it was a good problem that they’re not running CloudStrike antivirus.
It will be more complex in the future pinning any blame to anyone if they can’t get through to 999, as it could be the ISP, the AltNet, the power company or at the end of the line BT that had the issue.
Actually it’ll be really easy to find the cause of future outages on IP Networks, as all the Network Switches/Routers have the facility to log changes and faults, which providers will likely regularly download to assigned central servers. If something gets messed up there’ll be a record of it somewhere.
@Ex Telecom Engineer, a very good point. I guess it will depend whether the end user will ever get access though, likely they’ll all just point a finger and say “out of our control” such is the level of cust service we’ve come to expect from some of the players.
the complexity question has existed since 1985 or so with the launch of cellular phone networks, and only increased since as the cable companies, LLU and ITSPs all eroded BT’s telephone monopoly.
IIRC other telcos have been reprimanded for not doing their bit (faulty routing, not keeping the location database up to date, etc)
it’s just a reminder to people who think a PSTN line is more reliable than VOIP that in the event of emergency you’ll be dialling into a VOIP system anyway.
And that is the problem,
I have Voip here, not that I use it that often these days, but when I do it can be hit-and-miss and some of the call centres I have phoned over the years that use VoIP are awful. It as if we are going back to when the phone system was invented.
Get better VoIP. Many large enterprises have been using nothing else for years and it’s worked great for them. In a number of countries VoIP has been how the majority of people have been receiving and placing telephone services for a couple of decades. Poor quality lines are a feature of both digital and analogue calls.
I seem to remember that years ago it used to be a legal requirement that on a fixed landline that you had to have 1 handset that would still work in the event of a power cut. Now a lot of people I know who have migrated to FTTP or SOGEA have dropped the landline option altogether and just use mobiles.