Major Outage Strikes UK Biz Broadband and Hosting ISP Fastnet UPDATE3
Customers of business broadband and hosting provider Fastnet are currently experiencing a “major, wide scale service outage” across their VMWare hosting platform, which is affecting web, email, DNS hosting including cPanel services, Virtual Datacentre and some other services. But connectivity, DSL and Ethernet services are said to be “unaffected“.
At the time of writing, Fastnet’s service status page still appears to be offline and the ISP has instead replaced the front page of their website with a full-screen notice about the problem, which is rarely a good sign. The situation appears to have started last night at around 7-8pm and the vendor of their VMWare platform has been called in to assist with resolving it.
According to the most recent update, which was posted at 10:32am today: “Specialist engineers have arrived at our affected site and are progressing with their diagnosis, working as fast as possible to facilitate system recovery of our most essential services. We still cannot provide an ETA at this time.”
Advertisement
However, despite the ISP stating that their connectivity services are “unaffected“, the provider has also said: “Broadband customers that currently have working internet connections are advised NOT to reboot their routers in order to troubleshoot issues with our other services as this may cause subsequent reconnections to fail and take your connection offline.”
UPDATE 5:21pm
The most recent update, which was posted at 4:45pm, states: “Engineers are continuing work to activate the DR plan. We apologise for the impact to your services, and will keep this page updated with progress.”
At present there is still no indication of how long this will take to resolve, which must be quite worrying for their customers.
Advertisement
UPDATE 15th Aug 2024 @ 7:46am
The latest update, which was posted at 7:27am this morning, suggests that they’ve had to build some sort of new ‘recovery infrastructure’ to resolve the issue (the details are extremely vague): “The recovery infrastructure is now available, and engineers are working on data migration to the new environment.” But still no ETA for a resolution.
UPDATE 20th Aug 2024
The major outage appears to have finally been fully resolved on Saturday 17th August 2024, although some customers are reporting data loss and a change of passwords. In addition, the company has not yet updated their main Service Status page to reflect the incident.
Advertisement
Status Update 17/08/24 15:53
We are pleased to inform you that the majority of hosting services provided by Fastnet have now been restored following the recent incident affecting our VMware/Broadcom platform. Our teams have worked continuously to bring DNS, hosting and email services back online and fully operational. The remaining shared hosting platforms will be back online as soon as possible.
The root cause of the incident is still under investigation. Engineering teams are thoroughly analysing the situation, and we will continue to provide updates as we have more information.
Our connectivity and colocation services remain unaffected by this situation.
Mark is a professional technology writer, IT consultant and computer engineer from Dorset (England), he also founded ISPreview in 1999 and enjoys analysing the latest telecoms and broadband developments. Find me on X (Twitter), Mastodon, Facebook, BlueSky, Threads.net and Linkedin.
« Broadband ISP Onestream Boosts UK Online Security via NordVPN
From website: 14/08/24 13:59
Onsite engineers and vendor specialists are now working to prepare invocation of the DR plan. We expect a further update at 16:00
Oh dear, doesn’t sound good. Wonder if it’s a co-incidence this is just a day or so after VMWare dropped a bunch of critical updates. Hope that they can recover the service.
Interesting. Wonder if it’s an actual VMware issue or something like a SAN failure that has caused it.
Will never know! Used to be an infrastructure engineer and had to deal with incidents like this quite often.
Would not like to be them right now
The road to the RFO begins and ends with Hock Tan
It looks like something has properly trashed their VMware infrastructure. As you say, either a SAN issue or an update that went very very badly. Are the VMs being restored from backup or are they still accessible on the SAN?
The DNS is backup which is a relief. Our mail is with O365 so good news for us. Now look forward to the hosted web server restore. NOt sure my clinet has a backup. Something to look at!
We are a customer of theirs and our DNS zone is with them, it’s come backup in the last hour. We haven’t been given any more info than is on the status page.
Yep we have a number of web servers and SMTP relay services with Fastnet, been a client over 20 years, very knowledgeable team, but this is a massive fail , fear for the future of the business, the RFO will be interesting.. currently at 9pm 15th our services are still not available
We are a customer of theirs and we are still without email…
First noticed the problem around 18.00 on August 13.
It’s causing a serious loss of business for ourselves…
This is the third day without email—this is ridiculous. We had already planned to move somewhere else, and I’m frustrated that we haven’t done it yet. This is going to have a massive impact on their business.
As of 1549 today , Fri aug 16 it appears email is back ( at least receive is have not tested send yet)
All emails sent to me over the period of the outage are of course missing .
Some people have reported they got a address not found error when they tried
The latest update from their website
16/08/24 17:43
Our engineers have been working throughout the day restoring customer platforms including hosted VMs. A small subsection of shared customer platforms remain offline, and engineers are endeavouring to bring these back as soon as possible.
The root cause is still under investigation, and further works will continue this evening and into the weekend – more updates to follow.
Saturday – 10:30am Some of our web / email domains still non functional, this is a major catastrophe now unable to get updates or and communication, zero updates on Fastnet Twitter/X or FB accounts – very poor
Hi nick
Saturday 1842..
How is your situation now I wonder?
I have been with them over 21 years & this is the 1st time anything like this has happened .
Hope they fully diagnose the cause(s) soon & issue a full incident report
Latest update on Their website…
17/08/24 15:53
We are pleased to inform you that the majority of hosting services provided by Fastnet have now been restored following the recent incident affecting our VMware/Broadcom platform. Our teams have worked continuously to bring DNS, hosting and email services back online and fully operational. The remaining shared hosting platforms will be back online as soon as possible.
The root cause of the incident is still under investigation. Engineering teams are thoroughly analysing the situation, and we will continue to provide updates as we have more information.
Our connectivity and colocation services remain unaffected by this situation.
Once our investigations are complete, we will issue a full RFO to keep you informed.
We sincerely apologise for any inconvenience this has caused and thank you for your continued patience and understanding during this time.
My clients web data was restored by 9:30pm last night (Friday night). This was a relief as I could not find a way to re-direct the main domain DNS to an alternative server. Only sub domains. I have worked for MSPs myself and experienced similar issues at a data center. We had one data centre telling us they had an issue with a core switch when in fact the had gone bust. It will be interesting to hopefully fins out what actually happened with Fastnet. I believe they have done a good job with their DR plan in what was probably exceptional circumstances.
We’re a customer, and the outage took our system down for the better part of three days.
No communication from Fastnet during that time.
Just discovered today that the DB and all VMs have been rolled back nearly 24hrs meaning we’ve lost data.
No Communication From Fastnet on the data.
Now all the passwords have been changed “as a precaution”…
I sure hope this isn’t a prelude to a data breech disclosure.