Fluidata ISP Boss Talks Recent UK Internet Outages and Redundancy

The Managing Director of ISP Fluidata, Piers Daniell, has said that “lessons need to be learned about the importance of uptime and mitigating failure” after a big chunk of the UK suffered disruption to Internet services following power failures at two major London datacentres (here and here).

The affected datacentres (Telecity and Telehouse) happened to be two of the most popular sites in the UK, which meant that even a relatively brief power failure to key hardware was able to cause significant disruption across the country. Many ISPs were hit by the failure, although BT arguably suffered most of all due to the size of their network and customer base.

At this stage it’s unclear whether BT had sufficient alternative hardware in the datacentres so as to fully and quickly mitigate the problem, but Daniell suggests that “more attention to how these networks are built and what underpins the services” is perhaps just as important a buying decision as getting a great commercial deal.

Piers Daniell said:

“Precaution is key, and whether BT had or had not in place sufficient alternative hardware in that datacentre, it did have other working datacentres, so lessons need to be learned about the importance of uptime and mitigating failure where possible. Customers, and especially businesses and ISPs need to understand the risks of not just their networks but upstream suppliers to mitigate total outages.

Datacentres for example run large UPS (uninterruptable power supplies) systems to cope with the switch from mains power to generators and while this provides a level of resilience, it needs to be checked and serviced regularly. Servicing and testing varies dramatically between datacentres so investing in smaller UPS systems for individual racks may therefore seem excessive but from experience it can provide a useful buffer should the worse happen.

Furthermore the number of backups and spares again goes someway to reinforce confidence. Gone are the days, in my mind at least, when datacentres can offer n+1 resilience (where ‘n’ is the required load and the +1 means an additional spare). So for example if a datacentre requires four generators to power the site then five would be installed. This is very different to a more resilient site that offers 2n where, for the same example, eight generators would be provided. The big issue with all of this of course is cost and this has a knock-on effect to the customer.”

Fluidata’s MD correctly notes that a lot of the hardware inside datacentres and networks in general will spend more or less its whole life turned on and failures are bound to happen. “No datacentre is impervious to disaster, as BT and others experienced, and while a whole site outage is very rare, the importance of having multiple datacentres is very important,” said Daniell.

As always it often comes down to a question of cost. Yes you can have multiple datacentres and lots of redundancy, although this does become very expensive and complex. But as Daniell concludes, it also means the “likelihood of complete outage is seriously reduced.”

We can’t help but wonder if Ofcom’s plan to introduce a new system of automatic compensation for consumers might also make the desire for extra redundancy into a bigger requirement than it is now. Much will of course depend upon the detail of the regulator’s proposed approach and whether the outages of last week would even count.