Ericsson apologises for software fault that brought down cellular networks

• Mobile service outage in a reported 11 countries
• Caused by faulty software in Ericsson’s SGSN–MME packet switch equipment
• Likely culprit was an expired software certificate

Mea culpa. After Softbank laid the blame of this week’s network outage squarely at the door of Ericsson, its vendor partner has come out and apologised – not only for the disruption caused in Japan, but in a reported 11 other countries around the world (including the O2 network in the UK). Ericsson won’t comment on how many, but it reassures its CSP customers that it has taken immediate action to minimize impact and support the restoration of services.

The problems all occurred, simultaneously we believe, on Thursday December 6. An issue with certain nodes in core network equipment resulted in network disturbances for “a limited number of customers in multiple countries” using two specific software versions of the SGSN–MME (Serving GPRS Support Node – Mobility Management Entity). An initial root cause analysis indicates that the main issue was an expired certificate in the software versions installed with these customers. A complete and comprehensive root cause analysis is still in progress.

“The faulty software that has caused these issues is being decommissioned and we apologize not only to our customers but also to their customers,” said Börje Ekholm, President and CEO, Ericsson. “We work hard to ensure that our customers can limit the impact and restore their services as soon as possible.”

Oh to have been in earshot of the usually stoic Ekholm when the full impact of the news hit home. The phrase “heads will roll” comes to mind.

More detail on the situation as provided by Softbank. It announced that its LTE services of “SoftBank,” “Y!mobile,” Fixed Line Services called “Ouchi-no-Denwa” and a part of “SoftBank Air” services were not available for its use for over four hours on Thursday, “due to our network outage caused by Ericsson-made software errors related to its packet switches” covering its customers nationwide.

“We deeply apologize to our customers for all inconveniences it caused,” the CSP said in a public statement. “We will strive to take all measures to prevent the same network outage.”

SoftBank reports that its Network Center detected software malfunction in all of the packet switching equipment manufactured by Ericsson, which are installed at Tokyo Center and Osaka Center, and which cover its mobile customers nationwide. After the incident, SoftBank received a report from Ericsson that the software has been in operation for nine months ago and the failure caused by the same software also occurred simultaneously in other CSPs across 11 countries, which installed the same Ericsson-made devices. Softbank got its network back to normal operation by reinstalling the previous version of the software.

Losing its LTE service was bad enough, but Softbank also revealed the unwelcome knock-on effect of a corresponding data traffic surge in its 3G network that caused further difficulties for service use.

In our hyper-connected world, such outages have major implications. In just one of many, many local side-effects here in the UK, the local government Council in Angus, Scotland, reported that  “All parking meters in our off-street car parks are unavailable for use due to network issues.” Ah, the joys of cellular IoT and connect devices… still, we imagine that several hundred drivers would have been thrilled to enjoy a spot of free parking. And of course, there are plenty who will take this opportunity to try and claim compensation from their CSPs, who in turn will no doubt be shaking down their trusted vendor partner…