Openreach Interview – The Challenges of Delivering Fast UK Ethernet

Today high speed Ethernet lines supply capacity for everything from your broadband ISP connection to cash machines and delivering them can be difficult. So much so that Openreach’s (BT) Kevin Murphy has warned that some of Ofcom’s key performance targets may be “operationally unachievable.“

The term Ethernet is generally used to describe a family of computer networking technologies and topologies, which forms the foundation of both your Local Area Network (LAN) at home and many direct business connections. In this interview we’re focused more on the latter (i.e. high capacity data links between businesses and network operators).

Unlike your home connection, most Ethernet services tend to be dedicated private lines (fibre optic etc.) and these are usually installed at significant cost (thousands of pounds, sometimes tens of thousands etc.). Some of these connections can deliver transfer speeds that reach hundreds of Gigabits per second (Gbps) or even into multi-Terabit (Tbps) territory.

Naturally Openreach are one of the United Kingdom’s largest suppliers of Ethernet links, although in recent years their performance has been criticised. In response Ofcom imposed a series of Minimum Service Level (MSL) targets upon the operator, which were designed to improve performance.

Since then Openreach has made some impressive progress (e.g. their ability to fix 94.34% of repair jobs within 5 hours), although we recently reported that they were still falling short on two of the regulator’s primary performance targets (details) and the operator now believes that these might even be “operationally unachievable for reasons outside our control.”

Suffice to say that we wanted to know more about the challenges involved and what needs to change in order to tackle the current problems. Thankfully Openreach’s MD for Business and Corporate Delivery, Kevin Murphy, has been kind enough to engage with us on some of the core issues.

The Interview

Q1. Ethernet services tend to receive very little attention from the media and in that sense they are perhaps a bit misunderstood, yet they represent one of Openreach’s highest margin products. Could you tell our readers just a little bit about some of the key Ethernet products that Openreach provides and what they’re used for?

ANSWER:

Ethernet lines are the UK’s arteries – they provide the vital supporting network for countless digital services that we all enjoy and rely on every day.

If you’ve used a cash machine or paid for something by card or mobile in a shop; if you’ve ever posted something on Instagram or Twitter from your phone or streamed a song through Spotify, it’s pretty certain that data will have travelled over one of our Ethernet circuits at some point.

Ethernet lines connect thousands of our schools, hospitals, financial institutions, mobile masts and virtually every large business – and even TV, CCTV networks and data centres – so it’s no exaggeration to say that pretty much every aspect of daily life touches our Ethernet network in some way.

For our part, we cover an unrivalled geographic footprint in the UK and we offer a wide variety of high-bandwidth, permanently connected, point-to-point services designed to deliver a whole range of services to Communications Providers and their customers.

We offer a huge variety of services, but as an example there’s Ethernet Access Direct (EAD) – which provides speeds of 10Mbps to 10Gbps over a point-to-point data connection that can support a range of services from cloud computing to simultaneous online access for classrooms.

There’s also Ethernet Backhaul Direct (EBD) – a backhaul network stretching between 1,100 Openreach exchanges and offering 1Gbps and 10Gbps options. It’s a highly efficient, cost-effective way of delivering large volumes of data, quickly and securely, and it offers permanently connected, uncontended bandwidth which is proactively monitored, so we can pinpoint and fix problems in their infancy.

We’ll also soon be launching an alternative to ‘Dark Fibre’ called Optical Spectrum Access (OSA) Filter Connect. That will allow service providers to connect their own active equipment alongside Openreach-managed wavelengths and means they’ll be able to scale their bandwidth requirements without incurring additional cost, and without the service shortcomings of dark fibre

Q2. Ofcom set some tough business connectivity related quality and performance targets for Openreach’s Ethernet products and you’ve made some impressive progress in achieving those. Which of the Minimum Service Level (MSL) targets are you most proud of having hit or exceeded so far?

ANSWER:

We still want to do a lot more to improve service for our customers, but we are proud of what we’ve achieved, particularly on ‘Mean Time to Provide’ (MTTP) which is effectively the average speed of delivery.

We’ve also made a lot of progress on the ‘Lower Percentile’ MSL – the percentage of circuits that we deliver within six weeks – and on ‘Repair’, the percentage of faults that we fix within five hours, 24/7, 365 days a year. And we’re delivering more Ethernet circuits, faster than ever before.

When it comes to MTTP we are 40% faster than we were 2 years ago. I have no problem admitting that we were probably amongst the worst in Europe back then, but we’re now genuinely world class. I’m not aware of any operator in the world that’s delivering point-to-point fibre for businesses as quickly as we are.

And when we talk about repairs – we have hundreds of thousands of these (circuits) installed all over the UK ranging from shops on Oxford Street through to mobile cell sites in the Orkney Islands. If we get a fault on any of these lines, we fix more than 95 per cent of them within five hours, 24/7 & 365 days a year.

If you think of what that really means, it’s pretty astonishing. It means if something breaks in the middle of nowhere in rural Wales at 3am on a Sunday morning – there’s a 95 per cent chance of us fixing that problem in five hours. It’s an incredible feat of engineering.

Our customers and Ofcom have told us they’re noticing an improvement too, but we’re not stopping there.

Q3. As you may know, we recently reported that Openreach were at risk of missing two of the MSL targets. In particular, at the end of September 2017 some 7.56% of circuits took over 118 working days to deliver (target is <3%) and meanwhile 76.12% of circuits were being installed on the originally agreed delivery date (the target is 85% for 2017/18 and 90% for 2018/19).

The breakdown by region shows that the <3% target is suffering its biggest problem in Scotland, where 14.10% of circuits are now taking over 118 working days to install. Meanwhile it’s a similar story on the 85% target for circuit delivery, which is just 68.44% in Scotland and seems to be suffering a UK wide fall in performance.

Can you explain why Openreach appear to be struggling to achieve these specific two, particularly as such good progress has been made elsewhere?

ANSWER:

Those figures are for the three months up to the end of September 2017 so they’re four months out of date, and they typically relate to orders placed between January and February last year or even earlier – so it’s not really reflecting the current picture.

But you’re right that those specific measures are almost impossible to achieve due to reasons outside our control. We’ve been simultaneously fixing a tail of historic, complicated, overdue circuits, whilst increasing the overall volume of new orders that we deliver – and we’ve been pretty successful in doing that.

The frustrating thing is that the old orders only show up in the MSLs when they are closed, so there’s actually a perverse incentive for us not to tackle the ‘hard stuff’.

The MSL’s are tough and so have been a brilliant challenge to the team. We decided in 2016 to anchor all our improvements around the MSL’s, we have done the right thing by delivering the old, complex orders and in doing so this has whacked the performance stats. But we knew this would be a transition period on the way to our objective to having brilliant service for all our customers – which we are seeing through our Customer Satisfaction Scores which are up 45% in the year to date.

We believe the MSL’s drive the right behaviour and I say to my team that they will not define nor limit our ambitions to be the best, however as we clear away the historic circuits it has become clear that some of the measures are not achievable because of things like traffic noticing and permissions that run on our ‘Openreach clock’ and are outside our control.

We’re delivering more Ethernet circuits, faster than ever before and with a better understanding of our operational performance and we’ve submitted this evidence to Ofcom, so we’re hopeful they will re-consult on the MSLs, based on the evidence.

Over the last two years we’ve transformed Ethernet service and that has been recognised by our customers and by other key stakeholders including Ofcom.

Scotland presents some of the most challenging terrain in the country, particularly when you’re trying to deliver circuits within 40 days. We have some pretty exceptional challenges including having to lay sub-sea cables across the bottom of lochs, having to drill through solid granite to install cables and in one case we had to hire a helicopter to access a site in the Highlands & Islands to deliver a circuit (see pic below) – this is not Canary Wharf we’re talking about.

Q4. What are Openreach’s expectations for meeting these targets by the completion date and what changes may be required in order to achieve that outcome, if possible?

ANSWER:

Our service performance continues to improve. We’re connecting more Ethernet circuits and reducing the number of jobs in our workstack. On average we install Ethernet lines in around 37 days, and we fix 95% of Ethernet faults in 5 hours.

In July last year, the Competition Appeal Tribunal (CAT) ruled there were errors in Ofcom’s Business Connectivity Market Review market assessment and removed the Ethernet minimum service levels (MSLs). Ofcom then used emergency powers to re-impose remedies on us, including MSLs, covering the period up to the end of March 2019.

We’re currently outperforming four of the six temporary Ethernet MSLs, but we remain concerned that the two remaining MSLs are operationally unachievable for reasons outside our control, and we’re making further representations to Ofcom in relation to this.

In the meantime nothing changes in terms of our commitment to customers and service performance.

Q5. “The tail” – Orders over 118 working days (approx. 6 months) is at 7.56% against a target of 3%. In order to understand the scale of the issue, what are the absolute numbers (i.e. how many orders do you have that are older than 118 working days and are you able to reveal what is the oldest order)?

ANSWER:

The numbers are commercially confidential, but the percentage has come down significantly.

Continued over the page..