Mobile operator O2 (Telefonica) confirms that an unspecified “global software issue” has been identified as the cause of a major mobile data (mobile broadband) connectivity outage on their national UK 3G and 4G network this morning. Mobile providers in other countries are also believed to be affected.
The problems appear to have started just before 6am this morning, although contrary to O2’s statement we’ve noted that a sizeable number of their customers are also reporting difficulties with making calls and or sending text messages. In response the operator appears to be encouraging their customers to use WiFi “wherever they can” instead.
Unfortunately the outage also appears to be affecting customers of O2 based MVNO brands (e.g. Sky Mobile, Tesco Mobile, giffgaff and Lycamobile etc.) and any external services that rely on the operator’s data network (e.g. Transport for London’s bus timetable displays).
We’re aware that our customers are unable to use data this morning. One of our third party suppliers has identified a global software issue in their system which has impacted us. We believe other mobile operators around the world are also affected.
Our technical teams are working with their teams to ensure this is fixed as quickly as possible. We’d encourage our customers to use Wifi wherever they can and we apologise for the inconvenience caused.
End.
UPDATE 11:46am
O2 have issued a new status update, although it’s still saying broadly the same thing but with different wording.
O2 Status Update
The network issue we are currently experiencing remains a top priority for us and we are working as quickly as possible to fix the issues.
All O2 technical teams are working closely with one of our third party suppliers who has identified a global software issue in their system which has impacted data services.
We believe other mobile operators around the world are also affected.
We apologise to our customers for any inconvenience.
UPDATE 12:51pm
The fault appears to stem from one of Ericsson’s software systems.
UPDATE 7th Dec 2018
By the sounds of it 3G services were restored at about 6pm yesterday (Thursday) and 4G services were back online just after 3am this morning.
Tutela, the crowd-sourced mobile network analyst firm, has said their data suggests that over 90% of O2’s UK subscribers were affected and connectivity was “up and down” during the day, rather than just completely down (although most people will have experienced the latter).
Meanwhile O2 said their technical teams will continue to monitor service performance closely over the next few days and a “review will be carried out with Ericsson to understand fully what happened.”
Mark Evans, CEO of Telefonica (O2) UK, said:
“I want to let our customers know how sorry I am for the impact our network data issue has had on them, and reassure them that our teams, together with Ericsson, are doing everything we can.
We will continue to work with Ericsson, through the night, who have assured us that a full service will be restored for customers by the morning. We fully appreciate it’s been a poor experience and we are really sorry.”
Marielle Lindgren, CEO Ericsson UK & Ireland, added:
“The cause of today’s network issue is in certain nodes in the core network resulting in network disturbances for a limited number of customers across the world, including in the UK. We have been working hard on resolving the UK data issue since early this morning.
The faulty software that has caused these issues is being decommissioned. Our priority is to restore full data services on the network by tomorrow morning. Ericsson sincerely apologises to customers for the inconvenience caused.”
In our view this was the biggest mobile network outage that we’ve seen in the UK for at least the past decade. We can’t recall seeing anything quite as big as this, at least nothing that lasted for quite so long.
Given how bad their network is I suspect many customers didn’t notice.
Using o2 for work unfortunately. Whole office has sporadic data on their mobiles, causing massive issues as they’re used more for data than voice at work. Funny thing was I actually got 4g+ in the EU just yesterday.. then all of a sudden nothing, no data and only 2G and this was yesterday. Did also get over 200Mbit on 4G+ which was nice.
Considered getting a dual sim phone?
Well seeing as the places where I am has inside O2 coverage while the other do not – and in 3’s case no service at all I’d say for me anyway o2 has the best network.
Texts on Giffgaff are affected – all fail to send and I have not received any today.
Texts on GiffGaff appear to fail, and are retried, but in fact seem to get through. My wife received a reply to one that her phone had reported as failed.
My daughter’s watch is running on Giffgaff and I can’t get neither her GPS location not get/send message.
Here in West Devon, O2 4G signal is off air completely, 3G is still on air but no data connection, 2G is working for calls and SMS.
o2 4G + mobile internet is completely down here in West Dorset
#no2 !
Can’t even get a phone signal now in Falkirk never mind 3G/4G – what is going on o2?
Well I guess its a bit like you when your heating breaks down.
All the family can ask you as many times as they like – but all you can say is that I have phoned the heating person and they will be here to fix it as soon as they can.
You have no idea exactly why and what item has failed – and nor do O2
@Peter… the fault has existed since between 2 to 5AM today, so going by the time you posted your comment, at best they’ve had over 10 HOURS to locate the fault, that’s one crap service and technical team if at least 10 hours in they don’t know what the fault is!!!!!
@Mark, you have no idea what are you talking about.
@name.. actually yes I do.
It was a fault with Ericsson’s software.
They stated that early afternoon.
[They provide the infrastructure & software for the masts & cells]
guess we all get a days refund then…… NOT!!
I think Telefonica are having a siesta I received texts that apparently hadn’t sent at 13:00 but from 14:00 onwards have had no phone signal & inputting my details on the O2 status page results in a message saying the system is under too much pressure
Don’t worry they will fix their problems once they’ve sold your data
Had no Internet in Rotherham since 05:15 this morning canto access anything
I am sure that o2 / Ericsson are implementing the pre prepared roll back plan in case the up date failed.
Of course Ericsson will have tested the update prior on a pre production system.
However some one has dropped a major clanger here, however I think we can safely say someone(s) is getting sacked in the morning
Why would you fire someone who has just discovered a way in which O2 infrastructure can fail and an opportunity to fix and improve it?
People just seem so eager to get people fired these days.
IT teams are accused of running antiquated infrastructure partly because people are too scared of touching what isn’t broken no matter how fragile it may be getting.
TL;DR I doubt anyone’s getting fired.
Bugs and regressions of historical bugs make their way into production environments more frequently that you might imagine, even despite best efforts to prevent it. All big software/hardware vendors with solutions tjatve become increasingly more complex over time face this.
If the culture within these companies were to sack the people who screw up – they’d end up loosing a bunch of engineers and find it difficult to recruit new talent; say bye bye company…
I think most people subscribe to the idea that you hold your hands up, admit mistake and then compile a postmortem; understand the series of errors, mistakes or assumptions that lead a person or team to this position and work out how to prevent it again.
But, that’s just like, my opinion…
When do we get our refund?!
I haven’t had any data all day, on Tesco Mobile, and I struggled to make some phone calls! O2 / Ericsson won’t fix the issue till tomorrow morning…
I am now looking at EE’s deals, trouble is I like Tesco Mobile and it’s prices, but damn O2s network has had some major outages, more so any other.
Erricson said it was an expired certificate… Oppsy
Unfortunately, it’s usually 72hours / 3days of service issues we have to put up with before any MNO or home ISP will give a refund.
Texts on GiffGaff are still showing as ‘not sent’. Some are getting through, others not.
So Ericsson says their SSL certs expired. Amazing
https://www.ericsson.com/en/press-releases/2018/12/update-on-software-issue-impacting-certain-customers
If that is it well… the shame.
Sorry… No communicating with your community today. Software certificates have expired!!!
Just had a text from sky. I am getting unlimited data for the full day on the 8th of this month. These things happen. That seems like good compensation for me specifically. Others who use it for business, well, that a whole other matter.
Sky???
Yes, Sky. Sky use O2 network. Go to sky.com, it tells you on the front page.
Did you read the article 🙂
“Unfortunately the outage also appears to be affecting customers of O2 based MVNO brands (e.g. Sky Mobile, Tesco Mobile, giffgaff and Lycamobile etc.) and any external services that rely on the operator’s data network (e.g. Transport for London’s bus timetable displays”
meanwhile on Virgin Mobile and EE – all is well 🙂