Government Boosts Resilience of UK 999 Calls After BT Fault

The UK Government has today proposed a series of recommendations to help “bolster the resilience” of the 999 emergency call handling system, which is operated by BT, following a technical fault (“complex software issue”) last summer (here) that resulted in 11,470 unique emergency calls being “unsuccessfully connected“.

In case anybody has forgotten, BT previously revealed that a software bug had caused a “caching issue“, which resulted in impacted calls not being routed correctly and the user’s call being disconnected. A “robust temporary fix” was put in place to rectify this, and they’ve since implemented something more permanent.

The incident, the first in nearly 90 years, has now sparked the introduction of further safeguards to ensure continued public trust in the emergency call handling system, which has today been detailed as part of the Government’s post-incident review (here).

In terms of the changes, a new notification system between BT, the emergency services and UK Government has been put in place to “ensure all ambulance trusts and police forces” can rapidly enact a coordinated response. In the unlikely event of a future incident, the Government would also issue public advice on what to do to continue to reach the emergency services.

Howard Watson, Chief Security and Networks Officer, BT Group, said:

“At BT Group we take great pride in underpinning the national 999 service and recognise the critical importance our infrastructure plays. The level of disruption to the service on Sunday 25 June 2023 has never been seen before and we are sincerely sorry for the distress caused.

While no technology is 100% resilient, we have built a highly robust network with multiple layers of protection to connect the public to blue light services in their time of need. We take our responsibility to the emergency services and the public seriously, and on this occasion we fell short of our own high standards for the 999 service.

As outlined in the Government’s Post-Incident Review, we have put in place a comprehensive improvement plan to prevent this series of events reoccurring. We are also committed to working with all 999 stakeholders to continue to improve end-to-end resilience of the system.”

Michelle Donelan, Secretary of State for Science, Innovation and Technology, said:

“The incident in June of last year marked the first significant disruption to the 999 system in nearly 90 years. We are determined to prevent history from repeating itself, with public safety being absolutely paramount.

This is why, following a thorough review of the incident, we are working with BT to establish enhanced resilience measures, ensuring the UK is always prepared to effectively address major emergencies.

The government remains steadfast in its commitment to safeguarding the public’s safety and well-being.”

The post-incident review includes six recommendations (see below) and the most pressing recommendation (1) has already been completed, with BT implementing immediate improvements to its systems to prevent similar future occurrences. But the other recommendations aren’t due to be implemented until the end of April 2024 and one of them will take even longer.

The Six 999 Recommendations

Recommendation 1:

BT to strengthen its resilience of BT’s 999 platform, including the backup system and activation procedures. The procedure to activate the backup system failed on Sunday 25 June largely due to the complexity of the process and has now been simplified. In addition, the backup system provided more limited functionality and BT has committed to urgently improving the capabilities of the 999 platform, including the backup system, to ensure all information can be provided to the EAs when required (e.g. location data). A significant number of these improvements have already been made or are in the process of being implemented, and BT continues to provide regular progress updates on its improvement plans to all partners, whenever required. [COMPLETED]

Recommendation 2:

All EAs and BT ensure that risk management and business continuity plans remain appropriate to the range of risks to their operation of the 999 PECS, and that any insights relating to cross-cutting risks are shared with all partners of the 999 PECs end-to-end system. Acknowledging that this incident was only one fault within one part of the system, all EAs and BT should continue to strengthen the resilience of their respective areas to prepare for all risks that have the potential to disrupt the 999 PECS service, and share any findings with other PECS partners where impacts are likely to affect more than one area of the 999 PECS system. Where relevant, EAs and BT should draw on the expertise from technical authorities, such as the National Cyber Security Centre and National Protective Security Authority, consider the use of technical-level forums via the NICC or 999 Liaison Committee for cross-PECS discussions, and make reference to the 2023 National Risk Register. [ENDURING]

Recommendation 3:

HMG to establish a cross-PECS incident notification procedure and coordination group. Wherever possible, the principle of subsidiarity should also be maintained with partners empowered to take decisions at the operational level and coordination done at the highest necessary level. However, it was clear that the response to the 999 incident on 25 June was not as efficient as it could have been, and firm processes relating to notification, coordination, and communications have now been established. Following this principle and after collective agreement from all PECS partner on 17 July 2023, a nationwide 999 Strategic Incident protocol was established, consisting of: a formalised procedure to notify all relevant partners involved in the 999 end-to-end process in the right way within a reasonable timeframe; the provision of all appropriate contacts in all other organisations, with an established forum to convene at pace, if required; and aligned nationwide public messaging that can be issued in the event of an incident. An initial test of this new protocol was successfully completed on 14 July 2023, and it will be tested regularly as part of good practice. [COMPLETED]

Recommendation 4:

Cabinet Office to lead the coordination of all responsible departments in order to clarify existing responsibilities and accountability for the entire end-to-end PECS system and determine improved governance arrangements by the end of April 2024. Given the importance of 999 to UK citizen’s safety and security, and the number of delivery partners involved in the 999/112 PECS process, the Cabinet Office – in conjunction with all relevant departments – will increase oversight of the end-to-end resilience of the service, which will help all partners to support one another more effectively in strengthening the end-to-end resilience of the 999/112 PECS system, and in responding to any future incident. Any new governance arrangements for 999 will reflect the core principles for the UK’s overall approach to resilience as set out in the UK Government Resilience Framework, including ensuring that:

➤ there is a shared understanding of civil contingencies risks to the 999 PECS, and that risk management effort is appropriately balanced across the risk cycle – anticipation, assessment, prevention, mitigation, response, and recovery;
➤ responsibilities and accountabilities for 999 are clear, coordinated, and coherent (including ministerial responsibilities); and,
➤ the understanding of national and local risks is dynamic, driven by data and insight where appropriate, and informed by the best UK and international expertise and experience. [BY END APRIL 2024]

Recommendation 5:

Cabinet Office to coordinate with DHSC, DSIT, HO, DfT, DLUHC, DA’s, BT. communication teams to develop appropriate public advice that can be used for a number of scenarios involving disruption to 999, and ensure any advice is applicable across the UK by the end of April 2024. Plans will include an appropriate list of channels and key stakeholders to amplify the advice, so that this advice can quickly be tailored to the circumstances of any future incident and issued by the relevant authority. This deployment of this advice will be included in the exercising outlined in recommendation R6. [BY END APRIL 2024]

Recommendation 6:

Cabinet Office to coordinate an exercise to test the resilience of the 999 PECS system to future incidents, following completion of the above recommendations. This exercise should include appropriate representation from partners across the PECS system, such as LGDs for the various EAs (HO, DHSC, DfT), BT (and DSIT as its LGD), Cabinet Office, DLUHC, the EAs, and the DAs. The exercise should test:

➤ the effectiveness of the HMG 999 Response Framework in a range of PECS system disruption scenarios (established by R3);
➤ the roles and responsibilities of the above parties in a disruption to the PECS system against the principles of subsidiarity (established by R4); and,
➤ relevant public advice and clearance processes for any communication messaging during an incident (established by R5). [AFTER APRIL 2024]