Broadband ISP connections on CityFibre’s UK full fibre (FTTP) network in Glasgow have been knocked offline after a potential fire caused damage to the operator’s local Fibre Exchange (FEX) last night, which took down their core link (gla1215). But unusually they’re having to pull in a specialist electrician from Portsmouth as they don’t have any in Scotland.
The outage, which affected 1,936 of the provider’s local FTTP customers, appears to have started around 10:30pm following an “alert regarding high temperatures” at the Glasgow FEX. The operator’s engineers soon identified a “potential fire hazard” and immediately engaged the fire brigade, which then isolated the heat source from the power supply and cut power to the unit (hence why the core link went down).
The fire brigade then proceeded to ventilate the area and worked to lower the temperature, so engineers could safely access the exchange site. But by 5am this morning it was determined that the site “remains unsafe” and needed to be assessed by a fully qualified electrical engineer before CityFibre’s team could conduct their work.
Advertisement
However, none of the engineers on-site were found to be “fully qualified to perform the necessary assumptions or restoration work” (i.e. they need to bypass the UPS power, which has been cut, while the main power to the FEX is currently still active), which meant that the operator is having to pull in a qualified electrical engineer from as far away as Portsmouth on the south coast of England.
CityFibre Service Status Update (5am)
The vendor’s engineers on-site are not fully qualified to perform the necessary assumptions or restoration work. It has been confirmed that while the UPS power has been cut, the mains power to the FEX is still active. To safely bypass the UPS power, a qualified electrical engineer is required.
Unfortunately, the vendor currently has no qualified electrical engineers available in the Scottish region. They have assigned the task to an engineer based in Portsmouth, with an estimated time of arrival of 9:30 AM. All escalation avenues have been set with the vendor. Updates to follow pending the arrival of the Electrical Engineer.
In fairness, you often can’t just pull in any old electrician for this sort of job, which requires familiarity with the setup being used. But it is perhaps a little bit surprising, for an operator of their size, that they didn’t have any “qualified electrical engineers available” across the whole of the Scottish region or nearer than Portsmouth. At this stage it’s not known when the issue will be fully rectified, but that should hopefully occur sometime today.
Such exchanges form a vital part of CityFibre’s effort to cover up to 8 million UK premises with their Fibre-to-the-Premises (FTTP) network – representing c.30% of the UK. So far, they’ve covered around 3.8m premises and have connected 400,000 customers (8th May 2024), but are also known to be hunting for fresh investment (here). We have asked CityFibre to comment on this outage.
UPDATE 9:30am
Advertisement
We’ve had a comment from CityFibre, although it doesn’t add much to what is already known above. The operator’s engineering team are expected to issue another progress update within the hour, so hopefully that will bring a bit more news.
Paul Wakefield, Partnership Manager at CityFibre, said:
“An issue was identified last night at around 10.30pm, which has now been safely resolved. A specialist engineer is on site this morning to get everyone back online as soon as possible. We extend our apologies to those affected by the disruption.”
UPDATE 11am
Engineers are understood to have successfully restored power to the affected FEX in Glasgow, and services have been gradually coming back online following that work.
UPDATE 4:09pm
Advertisement
The most recent update states: “CityFibre continues to work on restoring services affected by the incident. Remote vendor support is actively working to recover the affected hardware. Additional field engineer resources are on-site, collaborating with CityFibre’s hardware vendors to restore the last remaining services.”
UPDATE 17th Oct 2024 @ 7:09am
CityFibre issued a short update last night to state that the “issue was fully resolved as of 17:05” yesterday afternoon. The operator has also since confirmed that the root cause was a “failure in the UPS (Uninterruptible Power Supply) battery system“, which sounds a lot like an awfully similar failure that impacted their FEX in Poole during early September last year. We can’t help but wonder if the same UPS/Battery vendor was being used for both.
Advertisement
Thanks for covering this. Hopefully it will stop some of the impacted from calling their ISP.
As one of the impacted.. Yea thanks.. I have been googling around to try to find out what’s borked, good to get some information.
Kind of annoying that neither CityFiber or our direct ISP Brillband have Status pages that customer can check for know outages. I hope they will take a lesson from this and implement a status page going forward.
I’m also with Brillband. I messaged them out of hours last night and got a reply around 23.30 saying they were aware of a localised fibre outage. I too am frustrated that there are no service status noticeboards or diagnostics on their website, or Cityfibre’s. Cityfibre just say to contact your provider, which is useless if it’s not within business hours. As much as I loathed Virgin, at least it was easy to find out what was happening during an outage.
If I was Ofcom I would be asking whether Cityfibre having only a single engineer in the country capable of resolving this is consistent with the General Condition of Entitlement clause A3.2:
“A3.2 Regulated Providers must take all necessary measures to ensure:
(a) the fullest possible availability of Voice Communications Services and Internet Access Services provided over Public Electronic Communications Networks in the event of catastrophic network breakdown or in cases of force majeure; and …” (the … is applicable to VoIP providers only).
Either Cityfibre should be expecting enforcement action, or Ofcom is failing in their duty.
Hahaha. Cityfibre provide neither service, voip nor Internet access.
So I think what you are saying is that rather than being directly regulated, it is Cityfibre’s resellers that are regulated?
That is easy then, Ofcom can fine Vodafone, Talktalk, Zen, Noone, Cuckoo, etc. etc. Sounds like more revenue for Ofcom…
It’s a bit of a grey area, but I think, as an infrastructure provider, CF are still obligated under Ofcom regulation to ensure a service is available.
Not sure if recovery from a fire at an exchange has specific Ofcom enforcment though. It’s probably covered by Force Majeur along with terrorism and natural disasters.
Cityfibre have a support contract with their equipment vendors. It’s the vendor who had no engineer available. I’m struggling to see that cityfibre have done anything wrong. Contract renewal time with the vendor might be interesting though.
one wonders how the recovery efforts might compare to a better resourced (some call it dinosaur/legacy) network operator that has customers and regulators demanding comparatively tight SLAs. Are these other firms perhaps too “efficient”?
Better, it’s already resolved.
They have 5 major cities covered in Glasgow and no one local in case of major failure – pathetic. I hope they managed to fly that poor sod who’s got to fix it up, 7 1/2 hours drive from Portsmouth to Glasgow is no joke.
It’s not actually Cityfibre’s fault so to speak. Their update suggests it’s an issue with a hardware vendor’s availability.
“Unfortunately, the vendor currently has no qualified electrical engineers available in the Scottish region. They have assigned the task to an engineer based in Portsmouth, with an estimated time of arrival of 9:30 AM. All escalation avenues have been set with the vendor. Updates to follow pending the arrival of the Electrical Engineer.”
Anon – for all we know, other, well-resourced operators might have had it resolved even sooner because they didn’t need to rely on one person to cover the entire country. One wonders what would have happened if that person was ill or on holiday.
ISP worker – I doubt we’ll ever get the exact detail but “vendor” could mean a lot of things, especially in an “efficient” organisation where everything is contracted out. Could be an equipment manufacturer, could be a facilities management firm.q
@ISP worker
CityFibre are ultimately responsible for the performance of their contractors, if they choose to contract out then it is up to them to make sure their supplier provides proper emergency coverage.
Nice to see BT Ivor and Small Mike have arrived to this article quicker than the CityFibre engineer did to fix the fault.
Cityfibre obviously have preferred contractors to carry out this kind of work, similar to VM having to rely on Eurocraft specialists for their VHub technologies should one ever get wiped out (in house staff can swap some parts). It’s really not unheard of…
Ivor, where does it say they are relying on one person to cover the entire country? Isn’t it far more likely this person was just the first available engineer? I mean you could argue they should pull a more local engineer off another job to get it fixed maybe a couple of hours earlier but the business has to weigh up the cost of doing that.
John – if the only person available to do this work is quite literally hundreds of miles away on the other side of the country, then there is a huge resource problem and I’d be highly concerned about the lack of resiliency if I was a CF ISP or customer.
I assume a “FEX” is roughly equivalent to a BT exchange. This is surely the highest priority and I’d assume that if there were any local staff available they’d already be in place.
Come on BT Ivor, what is good as “a business decision” quoted many times in favour/defence of BT, is surely good for competition, no?
It was an outage but fixed within reasonable time for residential.
CF anonymous, it would appear that it hasn’t been fully resolved – see other comments.
I don’t recall any instance where BT has been alleged to be understaffed and praised for it (certainly not by me)
I actually have had an OR outage recently – some kind of major fault at the cabinet (“dinosaur/legacy” FTTC) occured in the evening. Fixed overnight and BT’s 4G backup kicked in for the meantime. As it should be.
We’re seeing customers gradually coming back online here, too. I can’t go into specifics but about one third of affected customers have recovered in the last 45 mins.
Doesn’t help if you work from home and have lost over £100+ due to no internet service. Really isn’t good enough. Should be some kind of back up system in place. I mean City fibre did boast they were the future . It is them who maintain and run the tech that the ISP use to provide their service.
Yeah John, the consumer should really have a backup connection when working from home, especially when on a residential connection.
It’s stupid to assume these sort of connections are 100% up.
@John
If you run a business using a residential line or are dependent on Internet for your financial income perhaps it should be you that needs a back up?
Business lines have a much stricter SLA than residential… just a thought 🙂
No doubt John is not using a business fibre line with SLA’s, just a cheap domestic connection. Perhaps he could have a backup, such as a 4/5g mobile router for when you are working from home. They cost peanuts to buy and run. As an example, I have a mobile router with battery backup from TPLink, cost £60. The EE data sim subscription is £5/ month for 10Gb.
If it’s that critical to you, surely it’s worth arranging your own 5G backup system?
John – backup for customers is a second ISP on a different network if that important.
@John as a home worker myself I completely agree, it’s not ideal and there should be backups and contingency in place but I’m not sure how many eventualities a company can reasonably and financially be expected to prepare for.
Personally, I have my main Internet connection, then I have a SIM only plan with unlimited data so that if the home broadband goes down, I don’t have to commute to the office. Some say overkill, I say residency as I can be swapped over from one connection to another almost right away, compared to an hour to get to the office, by the time I’ve parked up and got to a desk etc.
@John, these things happen, that is the thing with technology it is not perfect and it goes wrong, a lot. I think they did well to get it all up and running again as fast as they did.
Openreach had no idea what was wrong when my FTTC service had problems a few years ago and told me to keep using the only thing that would connect, and that was an old Huawei modem.
Zzoomm had a problem the first few weeks I was with them and got it sorted, now over 12 months since then, all been fine.
If you work from home, then surly a back-up system is needed, even if it is just using a mobile phone to tether.
I’m in Glasgow…. And it’s still down
This is so inconvenience for businesses and students. Am staying in G21 my Internet is still down since Tuesday night..
Incident Start Time: 22:30 15/10/2024 Major Incident Manager: REMOVED NAME Summary Update: Following CityFibre NOC’s monitoring, it has been confirmed that 1,000 services have been successfully restored. However, ongoing issues are still affecting around 800 customers. We are now coordinating with the relevant teams to resolve these remaining service disruptions. In turn additional hardware and engineering support have been arranged at the Glasgow FEX to mitigate the current impact as quickly as possible. Completed Actions: – Additional Hardware and Engineering Support arranged. Next update at: 14:00 16/10/2025
A whole FEX is only serving 2000 customers. That’s either terrible planning or take up.
There are multiple Fex areas in Glasgow, the one affected is only partially complete. I would agree poor planning to an extent but Glasgow is a massive area.
I’m not surprised they don’t have many customers – I’m in Glasgow – I tried to get Cityfibre installed via Brawband – got notification my address was available for install and ordered in January 2024 – gave up in October after multiple cancelled or postponed or missed (just didn’t turn up on agreed date) appointments. I’ll stick with Three 5G – speeds around 600Mb down.
Issues weren’t Brawband’s fault (although they did start to charge before installation)
I was also concerned with the number of Cityfibre service interruption messages from Brawband, including today’s (although I cancelled, I still get service interruption messages)- seems to me that Cityfibre are an absolute shit show – poor reliability and poor communication to both end customers and the ISPs trying to use them.
I’m in Glasgow and Still DOWN has been since 1030 last night !!
Interesting, I wonder if this was UPS electronics or battery and its vendor.
Almost certainly. Is it a APC UPS…
In the last 20 years I had dozens if not more than 100 APC UPS devices, both at home and at work, rack and desktop. Only one rack UPS failed on hardware but it just turned off. I would be surprised if it was APC.
14:45. .. And in Glasgow…. Still waiting for reconnection
WELL 3.15PM STILL OFF NORTH GLASOW
Any update on this, my broadband is still down?
I’m in North West Glasgow and have had no service interruption – Vodafone via Cityfibre. My friend a few streets away has been offline since last night and he is Brillband via Cityfibre. Do Vodafone have their own infrastructure?
Any update on this my broadband is still down
Time to switch to BT then
Take a look at noone’s status page. It seems to regularly update
16:30 Still no internet service in North Glasgow. Been down since last night. Days wage lost. What is the update please.
Hi Jay, sorry to hear you’ve lost a day’s wages however if you rely on this connection so much maybe a business line would be more suitable with better SLA’s.
You’re using it for business reasons however paying residential prices.
4.43 pm North Glasgow still off. 4 lights on modem on instead of just 2 all day. So nearly there hopefully
16 48pm still down. All lights on modem on now. It was just 2 all day so hopefully soon
South Glasgow/ Lanarkshire area and connection was fine all day no outage – monitored on UDM Pro.
To anyone getting upset about this and complaining about lost business earnings. I would say that’s bad planning – backup solutions via starlink / 4/4g are fairly cheap in the scheme of things and if you rely on broadband as you suggest you do, then you should have a backup.
18:06 Still down. 4 lights on but still not connecting with the router.
still not working!!!
19.40pm internet still down for me unfortunately- north of Glasgow.
20:31 still down north Glasgow
Sadly, it’s what you expect when trigger happy executives go making people redundant without too much consideration or forward thinking
Then suddenly the experts aren’t there when you need them.
Given the maintenance of the kit in question isn’t handled by CityFibre I can’t see how their making you and others redundant could be responsible here.
Going by the service status contracts for this were potentially drawn up in 2018-19 as part of the FEX install contract.
@Witcher…
You sound like a CF spokesperson, easier to just blame a sub contractor.
Anyone else’s still down? Mine is still not working
Mine is still not working either in north of Glasgow
THURSDAY 17TH LUNCHTIME STILL DOWN SPOKE TO TALKTALK THEY SAY CITY FIBRE SAY ITS FIXED WELL MINE AINT
14:20 Still down Glasgow North
Located near Bearsden,Vodafone informed me it would be fixed by 3pm…it’s still not working.
are we all being treated like idiots still down
Still down north Glasgow with talk talk have all green lights on city fibre box now but still no internet connection frustrating almost 2 days now no internet TalkTalk useless seem to have no clue there is a fault in city fibre network
If it’s black talktalk router try a pinhole reset at back and wait 5 to 10 mins.
If it’s eero not sure
EERO MAY EERO RESET PRESS SMALL BUTTON UNDERNEATH BELOW SAN FRANCISCO
Thanks have tried the small button under San Francisco on eero still nothing city fibre where booked to call today and never turned up going round in circle with talk talk have you tried to unplug at the wall seems to be their answer for everything will be finding a new provider talk talk are useless
Same, they were meant to send an engineer round today and didn’t turn up, tried resetting, still not working.