A fault with an update issued by cybersecurity vendor CrowdStrike led to worldwide outages late last week. Planes were grounded around the world, banks and healthcare providers couldn’t service their clients, and entire businesses went offline.
“As we all know, CrowdStrike got caught pushing updates without regression testing,” says Harman Singh, director at Cyphere.
Recapping the issue, Singh says since CrowdStrike plants updates deep inside Windows, any faults corrupt the entire OS as well. This is what led to system restarting or displaying Windows’ infamous blue screen of death (BSOD).
CrowdStrike quickly identified the faulty update, and cobbled together a fix. But it’s been over five days and the service disruptions don’t seem to end. What gives?
No quick fix
Singh explains that the recovery methods that have been issued by CrowdStrike and Microsoft are very time consuming. They’d also require a significant amount of resources to roll out to the millions of affected computers.
Read | Microsoft outage: What is CrowdStrike Falcon and what does it do?
That said, he also points to CrowdStrike’s new approach, which he says has been a success.
“The reason everything isn’t back on track yet is that resolving the issue requires significant manual intervention,” says Yiyi Miao, chief product officer at OPSWAT.
This is further compounded by the fact that the systems that have been worst affected power critical infrastructure such as banks, hospitals and airlines.
“One of the key issues is that some of the impacted devices are mission-critical assets — used for keeping the business running, people’s life sustained and our country safe — so to suddenly put some of these systems out of commission for an unexpected maintenance cycle requires a significant change management plan to substitute in the backups, and sometimes there aren’t any backups,” explains Miao.
Secondly, Miao says there have been a lot of manual corrective approaches designed for encrypted systems. These systems were perhaps encrypted to comply with the data security standards at that organization.
Read: DXB airport restores operations after CrowdStrike-linked outage
“But when disaster recovery is needed on these systems, often the decryption keys are missing, which is a critical path to proceed with recovery,” says Miao.
Long road ahead
Alexey Lukatsky, managing director, cybersecurity business consultant at Positive Technologies, isn’t very fond of CrowdStrike’s remedial process. “In the absence of proven and tested update procedures, as well as due to the lack of IT specialists, delays are inevitable,” says Lukatsky.
He points that delays could also happen with organizations with a lot of remote workers. They’ll either have to bring in their devices, or follow company-defined recovery procedures, leading to further delays.
“The global impact of this IT issue makes it difficult to assess and set a quick recovery time,” says Singh.
CrowdStrike CEO George Kurtz has put out a statement apologizing for pushing out the erroneous update. He said that the company is “working closely with impacted customers and partners to ensure that all systems are restored”.
Given the number of affected computers, some are calling the CrowdStrike failure as the largest outage incident in history. Given the scope of the impacted computers, our experts believe the recovery process could take some time.
As things stand now, the CrowdStrike issue cannot be fixed remotely. Miao points out that currently it requires administrators to handle it on-site.
“Consequently, the resolution process could take days or even weeks,” says Miao. “The disruption from last week has caused a ripple effect, further delaying the restoration of normal operations.”
For more stories on tech, click here.
