ISPreview.co.uk Service Status Update – Downtime Apology

I’d just like to take a moment to apologise to our readers for ISPreview.co.uk’s recent downtime, which began late on Sunday night and lasted until Tuesday morning. Unfortunately the situation was out of our control and occurred when a fault struck just as we were in the middle of upgrading the web server.

A decision was taken last week to upgrade the server because the current hardware, especially the hard disk drive, was simply causing too many problems. It usually takes a couple of weeks for new hardware to be setup and made live and then the downtime is supposed to be almost unnoticeable (the old server would be mirrored until after the swap had been completed). On top of that we’re usually given warning about when this will happen.

Unfortunately in this instance nobody warned us that the host-side of the upgrade was due to take place on Sunday night, just as the majority of related front-line staff were being evacuated ahead of Hurricane Sandy. Yep, for some reason these are now based in the USA (east coast) instead of Europe where the physical server actually resides, go figure.

The lack of consultancy and admin staff meant nobody at the host was present to spot that the web server component wasn’t rebooting due to another fault and the mirrored site had similarly failed. After this it was just a matter of sitting around and twiddling our thumbs until the physical admin team resolved the fault and got the web server running again.

The last time we had an outage like this was about six years ago, it’s a rarity but such things do happen and in our experience nobody is immune. The devastation being broadcast in the media right now is a good indication of just why the necessary staff weren’t able to assist us, though we still don’t know why we had to go through the USA to get a problem with the EU server resolved. Multinational companies have unusual staffing structures.

Now back to work.. and oh god, there’s a lot of it.

PS – Thanks to all those many who flooded our inbox with offers of support and concern. Had no idea we’d be quite so missed 🙂 .