What Happened when CrowdStrike Lost It: A Critical Account of How Users Affected a Single Content Update for Windows Hosts

But that popularity put it in the position to wreak havoc when something went wrong, with systems using CrowdStrike and Windows-based hardware falling offline in droves this morning. CrowdStrike CEO George Kurtz said on Friday that the company is “actively working with customers impacted by a defect found in a single content update for Windows hosts” while emphasizing that the issue isn’t linked to a cyberattack. It does not affect Mac or Linux machines.

Hundreds of IT admins are reporting a lot of problems with their Windows machines, so there are a few steps to take, among which is to make sure the machines are in safe mode. That will be troublesome on some cloud-based servers, or even for Windows laptops that are deployed and used remotely.

The steps force Windows to boot into a Safe Mode environment if third-party drivers are not able to load. IT admins then have to locate the faulty driver on the disk and delete it. This workaround requires, in most cases, physical access to a machine. In some environments, it’s not easy to remove the faulty driver due to the use of BitLocker or lack of admin rights.

CrowdStrike’s update server and content delivery networks are likely being hammered by the millions of machines reaching its servers for an update, so it may take some time for the reboot method to work.

In that same interview, Kurtz apologized for the damage caused by CrowdStrike’s update, but there will undoubtedly be questions around how a faulty update like this ever managed to hit thousands or millions of machines around the world.

Notepad: An Update to CrowdStrike Anti-Spyware Platform for Monitoring and Mitigating the 2015-2016 Russian Cyberattacks

Notepad is a weekly newsletter that tracks the secrets and strategy behind Microsoft’s bets on artificial intelligence, gaming, and computing. Subscribe to get the latest straight to your inbox.

CrowdStrike plays an important role in helping companies find and prevent security breaches, billing itself as having the “fastest mean time” to detect threats. Since its launch in 2011, the Texas-based company has helped investigate major cyberattacks, such as the Sony Pictures hack in 2014, as well as the Russian cyberattacks on the Democratic National Committee in 2015 and 2016. CrowdStrike’s value was upwards of $83 billion as of Thursday evening.

The cause of one of the disasters is Buggy code, pushed out as an upgrade to CrowdStrike’sFalcon monitoring product, an anti-spyware platform that runs with deep system access on “endpoints” such as laptops, server, and routers to detect suspicious activity. CrowdStrike adds detections to the system to defend against new and evolving threats, so it requires permission to updateFalcon automatically and regularly. The downside of the arrangement is the danger that this system, which is meant to enhance security, may end up subverting it.

The update in question appears to have installed faulty software onto the core Windows operating system, causing systems to get stuck in a boot loop. Systems are showing an error message that says, “It looks like Windows didn’t load correctly,” while giving users the option to try troubleshooting methods or restart the PC. This airline in India uses the good old-fashioned way of doing things.

The Digital IoT Outage has Left a Globally Disrupted Business: The Case of American Airlines, Sky News, Ryanair, United, and Microsoft

“It reminds us about our dependence on IT and software,” Olejnik says. “When a system has several software systems maintained by various vendors, this is equivalent to placing trust on them. They may be a single point of failure—like here, when various firms feel the impact.”

The incident, so far, appears to only be impacting devices running Windows and not other operating systems. It is not clear how widespread the issues are or how long they will take to resolve. WIRED asked Microsoft and CrowdStrike if they had any comment on the outage.

Australian banks, airlines, and TV broadcasters first raised the alarm as thousands of machines started to go offline. The issues are now spreading, as businesses based in Europe are starting their working days. UK broadcaster Sky News is currently unable to broadcast its morning news bulletins, and is showing a message apologizing for “the interruption to this broadcast.” Ryanair, one of the biggest airlines in Europe, also says it’s experiencing a “third-party” IT issue, which is impacting flight departures.

The US Federal Aviation Administration says all flights from Delta, United, and American Airlines are grounded due to a “communication issue.” The Berlin airport warns of travel delays due to technical issues. Problems have impacted a lot of Alaska’s emergency call centers.

“Our entire company is offline” says one Reddit poster, while another says 70 percent of their laptops are down and stuck in a boot loop. One poster says happy Friday. It looks like it’s going to be a long day for IT admins worldwide.

In what appears to be a separate outage, Microsoft is also recovering from several issues with its Microsoft 365 apps and services. The root cause of those issues was a configuration change.

Banks, airports, TV stations, hotels, and countless other businesses are all facing widespread IT outages, leaving flights grounded and causing widespread disruption, after Windows machines have displayed errors worldwide.

Why hackers can’t do this: 12 hours of digital catastrophes aren’t triggered by software designed to destroy computers – they are happening now

A Microsoft spokesperson also issued a statement saying it is aware of the problems linked to Windows devices and the company believes a “resolution is forthcoming.”

The outage stemming from the CrowdStrike update has had a huge knock-on impact on public services and businesses around the world. One passenger in India is sharing a boarding pass with other airports that are facing delays and long lines.

Only a handful of times in history has a single piece of code managed to instantly wreck computer systems worldwide: The Slammer worm of 2003. Russia’s Ukraine-targeted NotPetya cyberattack. North Korea’s self-spreading ransomware WannaCry. But the ongoing digital catastrophe that rocked the internet and IT infrastructure worldwide over the last 12 hours appears to have been triggered not by malicious code released by hackers, but by the software designed to stop them.