A faulty software update issued by security giant CrowdStrike has resulted in a massive overnight outage that’s affected Windows computers around the world, disrupting businesses, airports, train stations, banks, broadcasters and the healthcare sector.
CrowdStrike said the outage was not caused by a cyberattack, but was the result of a “defect” in a software update for its flagship security product, Falcon Sensor. The defect caused any Windows computers that Falcon is installed on to crash without fully loading.
“The issue has been identified, isolated and a fix has been deployed,” said CrowdStrike in a statement on Friday. Some businesses and organizations are beginning to recover, but many expect the outages to drag on into the weekend or next week given the complexity of the fix. CrowdStrike CEO George Kurtz told NBC News that it may take “some time for some systems that just automatically won’t recover.” In a later tweet, Kurtz apologized for the disruption.
Here’s everything you need to know about the outages.
What happened?
Late Thursday into Friday, reports began to emerge of IT problems wherein Windows computers were getting stuck with the infamous “blue screen of death” — a bright blue error screen with a message that displays when Windows encounters a critical failure, crashes or cannot load.
The outages were first noticed in Australia early on Friday, and reports quickly came in from the rest of Asia and Europe as the regions began their day, as well as the United States.
Within a short time, CrowdStrike confirmed that a software update for Falcon had malfunctioned and was causing Windows computers that had the software installed to crash. Falcon lets CrowdStrike remotely analyze and check for malicious threats and malware on installed computers.
At around the same time, Microsoft reported a significant outage at one of its most used Azure cloud regions covering much of the central United States. A spokesperson for Microsoft told TechCrunch that its outage was unrelated to CrowdStrike’s incident.
Around Friday noon (Eastern time), Microsoft CEO Satya Nadella posted on X saying the company is aware of the CrowdStrike botched update and is “working closely with CrowdStrike and across the industry to provide customers technical guidance and support to safely bring their systems back online.”
What is CrowdStrike and what does Falcon Sensor do?
CrowdStrike, founded in 2011, has quickly grown into a cybersecurity giant. Today the company provides software and services to 29,000 corporate customers, including around half of Fortune 500 companies, 43 out of 50 U.S. states and eight out of the top 10 tech firms, according to its website.
The company’s cybersecurity software, Falcon, is used by enterprises to manage security on millions of computers around the world. These businesses include large corporations, hospitals, transportation hubs and government departments. Most consumer devices do not run Falcon and are unaffected by this outage.
One of the company’s biggest recent claims to fame was when it caught a group of Russian government hackers breaking into the Democratic National Committee ahead of the 2016 U.S. presidential election. CrowdStrike is also known for using memorable animal-themed names for the hacking groups it tracks based on their nationality, such as: Fancy Bear, believed to be part of Russia’s General Staff Main Intelligence Directorate, or GRU; Cozy Bear, believed to be part of Russia’s Foreign Intelligence Service, or SVR; Gothic Panda, believed to be a Chinese government group; and Charming Kitten, believed to be an Iranian state-backed group. The company even makes action figures to represent these groups, which it sells as swag.
CrowdStrike is so big it’s one of the sponsors of the Mercedes F1 team, and this year even aired a Super Bowl ad — a first for a cybersecurity company.
Who are the outages affecting?
Practically anyone who during their everyday life interacts with a computer system running software from CrowdStrike is affected, even if the computer isn’t theirs.
These devices include the cash registers at grocery stores, departure boards at airports and train stations, school computers, your work-issued laptops and desktops, airport check-in systems, airlines’ own ticketing and scheduling platforms, healthcare networks and many more. Because CrowdStrike’s software is so ubiquitous, the outages are causing chaos around the world in a variety of ways. A single affected Windows computer in a fleet of systems could be enough to disrupt the network.
TechCrunch reporters around the world are seeing and experiencing outages, including at points of travel, doctors’ offices and online. Early on Friday, the Federal Aviation Administration put in effect a ground stop, effectively grounding flights across the United States, citing the disruption. It looks like so far the national Amtrak rail network is functioning as normal.
What is the U.S. government doing so far?
Given that the problem stems from a company, there isn’t much that the U.S. federal government can do. According to a pool report, President Biden was briefed on the CrowdStrike outage, and “his team is in touch with CrowdStrike and impacted entities.” That’s in large part because the federal government is a customer of CrowdStrike and also affected.
Several federal agencies are affected by the incident, including the Department of Education, and Social Security Administration, which said Friday that it closed its offices as a result of the outage.
The pool report said Biden’s team is “engaged across the interagency to get sector by sector updates throughout the day and is standing by to provide assistance as needed.”
In a separate tweet, Homeland Security said it was working with its U.S. cybersecurity agency CISA, CrowdStrike and Microsoft — as well as its federal, state, local and critical infrastructure partners — to “fully assess and address system outages.”
There will no doubt be questions for CrowdStrike (and to some extent Microsoft, whose unrelated outage also caused disruption overnight for its customers) from government and congressional investigators.
For now, the immediate focus will be on the recovery of affected systems.
How do affected customers fix their Windows computers?
The major problem here is that CrowdStrike’s Falcon Sensor software malfunctioned, causing Windows machines to crash, and there’s no easy way to fix that.
So far, CrowdStrike has issued a patch, and it has also detailed a workaround that could help affected systems function normally until it has a permanent solution. One option is for users to “reboot the [affected computer] to give it an opportunity to download the reverted channel file,” referring to the fixed file.
In a message to users, CrowdStrike detailed a few steps customers can take, one of which requires physical access to an affected system to remove the defective file. CrowdStrike says users should boot the computer into Safe Mode or Windows Recovery Environment, navigate to the CrowdStrike directory, and delete the faulty file “C-00000291*.sys.”
The wider problem with having to fix the file manually could be a major headache for companies and organizations with large numbers of computers, or Windows-powered servers in datacenters or locations that might be in another region, or an entirely different country.
CISA warns that malicious actors are ‘taking advantage’ of the outage
In a statement on Friday, CISA attributed the outages to the faulty CrowdStrike update and that the issue was not due to a cyberattack. CISA said that it was “working closely with CrowdStrike and federal, state, local, tribal and territorial partners, as well as critical infrastructure and international partners to assess impacts and support remediation efforts.”
CISA did note, however, that it has “observed threat actors taking advantage of this incident for phishing and other malicious activity.” The cybersecurity agency did not provide more specifics, but warned organizations to stay vigilant.
Malicious actors can and will exploit confusion and chaos to carry out cyberattacks on their own. Rachel Tobac, a social engineering expert and founder of cybersecurity firm SocialProof Security, said in a series of posts on X to “verify people are who they say they are before taking sensitive actions.”
“Criminals will attempt to use this IT outage to pretend to be IT to you or you to IT to steal access, passwords, codes, etc.,” Tobac said.
What do we know about misinformation so far?
It’s easy to understand why some might have thought that this outage was a cyberattack. Sudden outages, blue screens at airports, office computers filled with error messages, and chaos and confusion. As you might expect, a fair amount of misinformation is already flying around, even as social media sites incorrectly flag trending topics like “cyberattack.”
Remember to check official sources of news and information, and if something seems too good to be true, it might just well be.
TechCrunch will keep this report updated throughout the day.
TechCrunch’s Ram Iyer contributed reporting.
Comment