Well, this sure is something to wake up to: a massive worldwide outage of computer systems due to a problem with CrowdStrike software. Payment systems, airlines, hospitals, governments, TV stations – pretty much anything or anyone using computers could be dealing with bluescreens, bootloops, and similar issues today. Open-heart surgeries had to be stopped mid-surgery, planes can’t take off, people can’t board trains, shoppers can’t pay for their groceries, and much, much more, all over the world.
The problem is caused by CrowdStrike, a sort-of enterprise AV/monitoring software that uses a Windows NT kernel driver to monitor everything people do on corporate machines and logs it for… Security purposes, I guess? I’ve never worked in a corporate setting so I have no experience with software like this. From what I hear, software like this is deeply loathed by workers the world over, as it gets in the way and slows systems down. And, as can happen with a kernel driver, a bug can cause massive worldwide outages which is costing people billions in damages and may even have killed people.
There is a workaround, posted by CrowdStrike:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
This is a solution for individually fixing affected machines, but I’ve seen responses like “great, how do I apply this to 70k endpoints?”, indicating that this may not be a practical solution for many affected customers. Then there’s the issue that this may require a BitLocker password, which not everyone has on hand either. To add insult to injury, CrowdStrike’s advisory about the issue is locked behind a login wall. A shitshow all around.
Do note that while the focus is on Windows, Linux machines can run CrowdStrike software too, and I’ve heard from Linux kernel engineers who happen to also administer large numbers of Linux servers that they’re seeing a huge spike in Linux kernel panics… Caused by CrowdStrike, which is installed on a lot more Linux servers than you might think. So while Windows is currently the focus of the story, the problems are far more widespread than just Windows.
I’m sure we’re going to see some major consequences here, and my – misplaced, I’m sure – hope is that this will make people think twice about one, using these invasive anti-worker monitoring tools, and two, employing kernel drivers for this nonsense.