Home
 » ISP News » 
Sponsored Links

CityFibre’s Broadband Services in Glasgow Downed by Potential Exchange Fire UPDATE4

Wednesday, Oct 16th, 2024 (8:25 am) - Score 7,360
CityFibre ODF in Fibre Exchange

Broadband ISP connections on CityFibre’s UK full fibre (FTTP) network in Glasgow have been knocked offline after a potential fire caused damage to the operator’s local Fibre Exchange (FEX) last night, which took down their core link (gla1215). But unusually they’re having to pull in a specialist electrician from Portsmouth as they don’t have any in Scotland.

The outage, which affected 1,936 of the provider’s local FTTP customers, appears to have started around 10:30pm following an “alert regarding high temperatures” at the Glasgow FEX. The operator’s engineers soon identified a “potential fire hazard” and immediately engaged the fire brigade, which then isolated the heat source from the power supply and cut power to the unit (hence why the core link went down).

NOTE: CityFibre is funded by c.£2.4bn in equity, c.£4.9bn debt and c.£800m of BDUK / public subsidy. The company is owned by Antin Infrastructure Partners, Goldman Sachs Asset Management, Mubadala Investment Company and Interogo Holding. Multiple ISPs sell related packages on their network (e.g. Vodafone, TalkTalk, Zen Internet and soon Sky Broadband etc.).

The fire brigade then proceeded to ventilate the area and worked to lower the temperature, so engineers could safely access the exchange site. But by 5am this morning it was determined that the site “remains unsafe” and needed to be assessed by a fully qualified electrical engineer before CityFibre’s team could conduct their work.

Advertisement

However, none of the engineers on-site were found to be “fully qualified to perform the necessary assumptions or restoration work” (i.e. they need to bypass the UPS power, which has been cut, while the main power to the FEX is currently still active), which meant that the operator is having to pull in a qualified electrical engineer from as far away as Portsmouth on the south coast of England.

CityFibre Service Status Update (5am)

The vendor’s engineers on-site are not fully qualified to perform the necessary assumptions or restoration work. It has been confirmed that while the UPS power has been cut, the mains power to the FEX is still active. To safely bypass the UPS power, a qualified electrical engineer is required.

Unfortunately, the vendor currently has no qualified electrical engineers available in the Scottish region. They have assigned the task to an engineer based in Portsmouth, with an estimated time of arrival of 9:30 AM. All escalation avenues have been set with the vendor. Updates to follow pending the arrival of the Electrical Engineer.

In fairness, you often can’t just pull in any old electrician for this sort of job, which requires familiarity with the setup being used. But it is perhaps a little bit surprising, for an operator of their size, that they didn’t have any “qualified electrical engineers available” across the whole of the Scottish region or nearer than Portsmouth. At this stage it’s not known when the issue will be fully rectified, but that should hopefully occur sometime today.

Such exchanges form a vital part of CityFibre’s effort to cover up to 8 million UK premises with their Fibre-to-the-Premises (FTTP) network – representing c.30% of the UK. So far, they’ve covered around 3.8m premises and have connected 400,000 customers (8th May 2024), but are also known to be hunting for fresh investment (here). We have asked CityFibre to comment on this outage.

UPDATE 9:30am

Advertisement

We’ve had a comment from CityFibre, although it doesn’t add much to what is already known above. The operator’s engineering team are expected to issue another progress update within the hour, so hopefully that will bring a bit more news.

Paul Wakefield, Partnership Manager at CityFibre, said:

“An issue was identified last night at around 10.30pm, which has now been safely resolved. A specialist engineer is on site this morning to get everyone back online as soon as possible. We extend our apologies to those affected by the disruption.”

UPDATE 11am

Engineers are understood to have successfully restored power to the affected FEX in Glasgow, and services have been gradually coming back online following that work.

UPDATE 4:09pm

Advertisement

The most recent update states: “CityFibre continues to work on restoring services affected by the incident. Remote vendor support is actively working to recover the affected hardware. Additional field engineer resources are on-site, collaborating with CityFibre’s hardware vendors to restore the last remaining services.”

UPDATE 17th Oct 2024 @ 7:09am

CityFibre issued a short update last night to state that the “issue was fully resolved as of 17:05” yesterday afternoon. The operator has also since confirmed that the root cause was a “failure in the UPS (Uninterruptible Power Supply) battery system“, which sounds a lot like an awfully similar failure that impacted their FEX in Poole during early September last year. We can’t help but wonder if the same UPS/Battery vendor was being used for both.

Share with Twitter
Share with Linkedin
Share with Facebook
Share with Reddit
Share with Pinterest
Tags: , ,
Mark-Jackson
By Mark Jackson
Mark is a professional technology writer, IT consultant and computer engineer from Dorset (England), he also founded ISPreview in 1999 and enjoys analysing the latest telecoms and broadband developments. Find me on X (Twitter), Mastodon, Facebook and .
Search ISP News
Search ISP Listings
Search ISP Reviews
Comments
69 Responses

Advertisement

  1. Avatar photo Rik says:

    Thanks for covering this. Hopefully it will stop some of the impacted from calling their ISP.

  2. Avatar photo Natasha says:

    As one of the impacted.. Yea thanks.. I have been googling around to try to find out what’s borked, good to get some information.

    Kind of annoying that neither CityFiber or our direct ISP Brillband have Status pages that customer can check for know outages. I hope they will take a lesson from this and implement a status page going forward.

    1. Avatar photo Graham says:

      I’m also with Brillband. I messaged them out of hours last night and got a reply around 23.30 saying they were aware of a localised fibre outage. I too am frustrated that there are no service status noticeboards or diagnostics on their website, or Cityfibre’s. Cityfibre just say to contact your provider, which is useless if it’s not within business hours. As much as I loathed Virgin, at least it was easy to find out what was happening during an outage.

  3. Avatar photo Dassa says:

    If I was Ofcom I would be asking whether Cityfibre having only a single engineer in the country capable of resolving this is consistent with the General Condition of Entitlement clause A3.2:

    “A3.2 Regulated Providers must take all necessary measures to ensure:
    (a) the fullest possible availability of Voice Communications Services and Internet Access Services provided over Public Electronic Communications Networks in the event of catastrophic network breakdown or in cases of force majeure; and …” (the … is applicable to VoIP providers only).

    Either Cityfibre should be expecting enforcement action, or Ofcom is failing in their duty.

    1. Avatar photo H says:

      Hahaha. Cityfibre provide neither service, voip nor Internet access.

    2. Avatar photo Dassa says:

      So I think what you are saying is that rather than being directly regulated, it is Cityfibre’s resellers that are regulated?

      That is easy then, Ofcom can fine Vodafone, Talktalk, Zen, Noone, Cuckoo, etc. etc. Sounds like more revenue for Ofcom…

    3. Avatar photo Simon says:

      It’s a bit of a grey area, but I think, as an infrastructure provider, CF are still obligated under Ofcom regulation to ensure a service is available.
      Not sure if recovery from a fire at an exchange has specific Ofcom enforcment though. It’s probably covered by Force Majeur along with terrorism and natural disasters.

    4. Avatar photo 125us says:

      Cityfibre have a support contract with their equipment vendors. It’s the vendor who had no engineer available. I’m struggling to see that cityfibre have done anything wrong. Contract renewal time with the vendor might be interesting though.

  4. Avatar photo Ivor says:

    one wonders how the recovery efforts might compare to a better resourced (some call it dinosaur/legacy) network operator that has customers and regulators demanding comparatively tight SLAs. Are these other firms perhaps too “efficient”?

    1. Avatar photo Anon says:

      Better, it’s already resolved.

    2. Avatar photo Big Dave says:

      They have 5 major cities covered in Glasgow and no one local in case of major failure – pathetic. I hope they managed to fly that poor sod who’s got to fix it up, 7 1/2 hours drive from Portsmouth to Glasgow is no joke.

    3. Avatar photo ISP worker says:

      It’s not actually Cityfibre’s fault so to speak. Their update suggests it’s an issue with a hardware vendor’s availability.

      “Unfortunately, the vendor currently has no qualified electrical engineers available in the Scottish region. They have assigned the task to an engineer based in Portsmouth, with an estimated time of arrival of 9:30 AM. All escalation avenues have been set with the vendor. Updates to follow pending the arrival of the Electrical Engineer.”

    4. Avatar photo Ivor says:

      Anon – for all we know, other, well-resourced operators might have had it resolved even sooner because they didn’t need to rely on one person to cover the entire country. One wonders what would have happened if that person was ill or on holiday.

      ISP worker – I doubt we’ll ever get the exact detail but “vendor” could mean a lot of things, especially in an “efficient” organisation where everything is contracted out. Could be an equipment manufacturer, could be a facilities management firm.q

    5. Avatar photo Big Dave says:

      @ISP worker

      CityFibre are ultimately responsible for the performance of their contractors, if they choose to contract out then it is up to them to make sure their supplier provides proper emergency coverage.

    6. Avatar photo IvorMike says:

      Nice to see BT Ivor and Small Mike have arrived to this article quicker than the CityFibre engineer did to fix the fault.

      Cityfibre obviously have preferred contractors to carry out this kind of work, similar to VM having to rely on Eurocraft specialists for their VHub technologies should one ever get wiped out (in house staff can swap some parts). It’s really not unheard of…

    7. Avatar photo john says:

      Ivor, where does it say they are relying on one person to cover the entire country? Isn’t it far more likely this person was just the first available engineer? I mean you could argue they should pull a more local engineer off another job to get it fixed maybe a couple of hours earlier but the business has to weigh up the cost of doing that.

    8. Avatar photo Ivor says:

      John – if the only person available to do this work is quite literally hundreds of miles away on the other side of the country, then there is a huge resource problem and I’d be highly concerned about the lack of resiliency if I was a CF ISP or customer.

      I assume a “FEX” is roughly equivalent to a BT exchange. This is surely the highest priority and I’d assume that if there were any local staff available they’d already be in place.

    9. Avatar photo anonymous says:

      Come on BT Ivor, what is good as “a business decision” quoted many times in favour/defence of BT, is surely good for competition, no?

      It was an outage but fixed within reasonable time for residential.

    10. Avatar photo Ivor says:

      CF anonymous, it would appear that it hasn’t been fully resolved – see other comments.

      I don’t recall any instance where BT has been alleged to be understaffed and praised for it (certainly not by me)

      I actually have had an OR outage recently – some kind of major fault at the cabinet (“dinosaur/legacy” FTTC) occured in the evening. Fixed overnight and BT’s 4G backup kicked in for the meantime. As it should be.

  5. Avatar photo ISP worker says:

    We’re seeing customers gradually coming back online here, too. I can’t go into specifics but about one third of affected customers have recovered in the last 45 mins.

    1. Avatar photo John. says:

      Doesn’t help if you work from home and have lost over £100+ due to no internet service. Really isn’t good enough. Should be some kind of back up system in place. I mean City fibre did boast they were the future . It is them who maintain and run the tech that the ISP use to provide their service.

    2. Avatar photo Carlos says:

      Yeah John, the consumer should really have a backup connection when working from home, especially when on a residential connection.

      It’s stupid to assume these sort of connections are 100% up.

    3. Avatar photo Anon says:

      @John

      If you run a business using a residential line or are dependent on Internet for your financial income perhaps it should be you that needs a back up?

      Business lines have a much stricter SLA than residential… just a thought 🙂

    4. Avatar photo Andrew S says:

      No doubt John is not using a business fibre line with SLA’s, just a cheap domestic connection. Perhaps he could have a backup, such as a 4/5g mobile router for when you are working from home. They cost peanuts to buy and run. As an example, I have a mobile router with battery backup from TPLink, cost £60. The EE data sim subscription is £5/ month for 10Gb.

    5. Avatar photo A Stevens says:

      If it’s that critical to you, surely it’s worth arranging your own 5G backup system?

    6. Avatar photo The Facts says:

      John – backup for customers is a second ISP on a different network if that important.

    7. Avatar photo ISP worker says:

      @John as a home worker myself I completely agree, it’s not ideal and there should be backups and contingency in place but I’m not sure how many eventualities a company can reasonably and financially be expected to prepare for.

      Personally, I have my main Internet connection, then I have a SIM only plan with unlimited data so that if the home broadband goes down, I don’t have to commute to the office. Some say overkill, I say residency as I can be swapped over from one connection to another almost right away, compared to an hour to get to the office, by the time I’ve parked up and got to a desk etc.

    8. Avatar photo Ad47uk says:

      @John, these things happen, that is the thing with technology it is not perfect and it goes wrong, a lot. I think they did well to get it all up and running again as fast as they did.
      Openreach had no idea what was wrong when my FTTC service had problems a few years ago and told me to keep using the only thing that would connect, and that was an old Huawei modem.

      Zzoomm had a problem the first few weeks I was with them and got it sorted, now over 12 months since then, all been fine.

      If you work from home, then surly a back-up system is needed, even if it is just using a mobile phone to tether.

  6. Avatar photo Craig Glover says:

    I’m in Glasgow…. And it’s still down

    1. Avatar photo Sukai Jallow says:

      This is so inconvenience for businesses and students. Am staying in G21 my Internet is still down since Tuesday night..

  7. Avatar photo ISP says:

    Incident Start Time: 22:30 15/10/2024 Major Incident Manager: REMOVED NAME Summary Update: Following CityFibre NOC’s monitoring, it has been confirmed that 1,000 services have been successfully restored. However, ongoing issues are still affecting around 800 customers. We are now coordinating with the relevant teams to resolve these remaining service disruptions. In turn additional hardware and engineering support have been arranged at the Glasgow FEX to mitigate the current impact as quickly as possible. Completed Actions: – Additional Hardware and Engineering Support arranged. Next update at: 14:00 16/10/2025

  8. Avatar photo N says:

    A whole FEX is only serving 2000 customers. That’s either terrible planning or take up.

    1. Avatar photo Anon says:

      There are multiple Fex areas in Glasgow, the one affected is only partially complete. I would agree poor planning to an extent but Glasgow is a massive area.

    2. Avatar photo Iain Harris says:

      I’m not surprised they don’t have many customers – I’m in Glasgow – I tried to get Cityfibre installed via Brawband – got notification my address was available for install and ordered in January 2024 – gave up in October after multiple cancelled or postponed or missed (just didn’t turn up on agreed date) appointments. I’ll stick with Three 5G – speeds around 600Mb down.

      Issues weren’t Brawband’s fault (although they did start to charge before installation)

      I was also concerned with the number of Cityfibre service interruption messages from Brawband, including today’s (although I cancelled, I still get service interruption messages)- seems to me that Cityfibre are an absolute shit show – poor reliability and poor communication to both end customers and the ISPs trying to use them.

  9. Avatar photo J says:

    I’m in Glasgow and Still DOWN has been since 1030 last night !!

  10. Avatar photo Name says:

    Interesting, I wonder if this was UPS electronics or battery and its vendor.

    1. Avatar photo - says:

      Almost certainly. Is it a APC UPS…

    2. Avatar photo Name says:

      In the last 20 years I had dozens if not more than 100 APC UPS devices, both at home and at work, rack and desktop. Only one rack UPS failed on hardware but it just turned off. I would be surprised if it was APC.

  11. Avatar photo Craig says:

    14:45. .. And in Glasgow…. Still waiting for reconnection

  12. Avatar photo thomas scullion says:

    WELL 3.15PM STILL OFF NORTH GLASOW

  13. Avatar photo Alex says:

    Any update on this, my broadband is still down?

  14. Avatar photo Doofus says:

    I’m in North West Glasgow and have had no service interruption – Vodafone via Cityfibre. My friend a few streets away has been offline since last night and he is Brillband via Cityfibre. Do Vodafone have their own infrastructure?

  15. Avatar photo W says:

    Any update on this my broadband is still down

    1. Avatar photo CFsound says:

      Time to switch to BT then

    2. Avatar photo Robin says:

      Take a look at noone’s status page. It seems to regularly update

  16. Avatar photo Jay says:

    16:30 Still no internet service in North Glasgow. Been down since last night. Days wage lost. What is the update please.

    1. Avatar photo Anon says:

      Hi Jay, sorry to hear you’ve lost a day’s wages however if you rely on this connection so much maybe a business line would be more suitable with better SLA’s.

      You’re using it for business reasons however paying residential prices.

  17. Avatar photo thomas scullion says:

    4.43 pm North Glasgow still off. 4 lights on modem on instead of just 2 all day. So nearly there hopefully

  18. Avatar photo thomas scullion says:

    16 48pm still down. All lights on modem on now. It was just 2 all day so hopefully soon

  19. Avatar photo John says:

    South Glasgow/ Lanarkshire area and connection was fine all day no outage – monitored on UDM Pro.

  20. Avatar photo Andrew says:

    To anyone getting upset about this and complaining about lost business earnings. I would say that’s bad planning – backup solutions via starlink / 4/4g are fairly cheap in the scheme of things and if you rely on broadband as you suggest you do, then you should have a backup.

  21. Avatar photo J says:

    18:06 Still down. 4 lights on but still not connecting with the router.

  22. Avatar photo Alex Melrose says:

    still not working!!!

  23. Avatar photo Liz says:

    19.40pm internet still down for me unfortunately- north of Glasgow.

  24. Avatar photo Kevin O'Neill says:

    20:31 still down north Glasgow

  25. Avatar photo DF says:

    Sadly, it’s what you expect when trigger happy executives go making people redundant without too much consideration or forward thinking

    Then suddenly the experts aren’t there when you need them.

    1. Avatar photo Witcher says:

      Given the maintenance of the kit in question isn’t handled by CityFibre I can’t see how their making you and others redundant could be responsible here.

      Going by the service status contracts for this were potentially drawn up in 2018-19 as part of the FEX install contract.

    2. Avatar photo DF says:

      @Witcher…

      You sound like a CF spokesperson, easier to just blame a sub contractor.

  26. Avatar photo Heather says:

    Anyone else’s still down? Mine is still not working

    1. Avatar photo Liz says:

      Mine is still not working either in north of Glasgow

  27. Avatar photo Thomas says:

    THURSDAY 17TH LUNCHTIME STILL DOWN SPOKE TO TALKTALK THEY SAY CITY FIBRE SAY ITS FIXED WELL MINE AINT

  28. Avatar photo Kevin O'Neill says:

    14:20 Still down Glasgow North

    1. Avatar photo Alan says:

      Located near Bearsden,Vodafone informed me it would be fixed by 3pm…it’s still not working.

  29. Avatar photo thomas scullion says:

    are we all being treated like idiots still down

  30. Avatar photo Mark says:

    Still down north Glasgow with talk talk have all green lights on city fibre box now but still no internet connection frustrating almost 2 days now no internet TalkTalk useless seem to have no clue there is a fault in city fibre network

    1. Avatar photo Anon says:

      If it’s black talktalk router try a pinhole reset at back and wait 5 to 10 mins.

      If it’s eero not sure

  31. Avatar photo thomas scullion says:

    EERO MAY EERO RESET PRESS SMALL BUTTON UNDERNEATH BELOW SAN FRANCISCO

  32. Avatar photo Mark says:

    Thanks have tried the small button under San Francisco on eero still nothing city fibre where booked to call today and never turned up going round in circle with talk talk have you tried to unplug at the wall seems to be their answer for everything will be finding a new provider talk talk are useless

    1. Avatar photo Heather says:

      Same, they were meant to send an engineer round today and didn’t turn up, tried resetting, still not working.

Leave a Reply

Your email address will not be published. Required fields are marked *

NOTE: Your comment may not appear instantly (it may take several hours) due to static caching or random moderation checks by the anti-spam system.
Javascript must be enabled to post (most browsers do this automatically)

Privacy Notice: Please note that news comments are anonymous, which means that we do NOT require you to enter any real personal details to post a message. By clicking to submit a post you agree to storing your comment content, display name, IP, email and / or website details in our database, for as long as the post remains live.

Only the submitted name and comment will be displayed in public, while the rest will be kept private (we will never share this outside of ISPreview, regardless of whether the data is real or fake). This comment system uses submitted IP, email and website address data to spot abuse and spammers. All data is transferred via an encrypted (https secure) session.

NOTE 1: Sometimes your comment might not appear immediately due to site cache (this is cleared every few hours) or it may be caught by automated moderation / anti-spam.

NOTE 2: Comments that break our rules, spam, troll or post via known fake IP/proxy servers may be blocked or removed.
Cheap BIG ISPs for 100Mbps+
Community Fibre UK ISP Logo
150Mbps
Gift: None
Vodafone UK ISP Logo
Vodafone £24.00 - 26.00
150Mbps
Gift: None
NOW UK ISP Logo
NOW £25.00
100Mbps
Gift: None
Virgin Media UK ISP Logo
Virgin Media £25.00
132Mbps
Gift: None
Hyperoptic UK ISP Logo
Hyperoptic £27.50
158Mbps
Gift: None
Large Availability | View All
Cheapest ISPs for 100Mbps+
Gigaclear UK ISP Logo
Gigaclear £17.00
200Mbps
Gift: None
Brsk UK ISP Logo
Brsk £19.00
150Mbps
Gift: None
Community Fibre UK ISP Logo
150Mbps
Gift: None
Hey! Broadband UK ISP Logo
150Mbps
Gift: None
YouFibre UK ISP Logo
YouFibre £23.99
150Mbps
Gift: None
Large Availability | View All
The Top 15 Category Tags
  1. FTTP (5965)
  2. BT (3628)
  3. Politics (2692)
  4. Business (2416)
  5. Openreach (2397)
  6. Building Digital UK (2320)
  7. Mobile Broadband (2115)
  8. FTTC (2075)
  9. Statistics (1883)
  10. 4G (1785)
  11. Virgin Media (1740)
  12. Ofcom Regulation (1567)
  13. Fibre Optic (1461)
  14. Wireless Internet (1452)
  15. FTTH (1385)
Promotion
Sponsored

Copyright © 1999 to Present - ISPreview.co.uk - All Rights Reserved - Terms , Privacy and Cookie Policy , Links , Website Rules , Contact
Mastodon