CrowdStrike’s Update Causes Global IT Outage, Resulting in Windows Blue Screen of Death and Causing a Worldwide Storm
The world is more connected than ever before. With technology unfolding and intertwining across continents and nations to keep us in communication, to keep our businesses running, and to keep our data safe. Unfortunately, these connections also mean that damage to one end of the network will affect millions. This is exactly how things have unfolded following the global IT outage that caused the Blue Screen of Death BSOD to appear on the screens of several large organizations across the world. In this blog article, we explore the details of the CrowdStrike outage and subsequent Microsoft outage, the impact the incident had around the globe, and how these companies are responding. We also look into what have caused the IT global outage.
CrowdStrike's Faulty Update Makes Waves
On the morning of the 19th of July, Microsoft users turned on their computers to find themselves facing the Blue Screen of Death BSOD. Microsoft noted that it was aware of the issue “impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon Sensor, which may encounter a bug check – or Blue Screen of Death - and get stuck in a restarting state.” Cybersecurity firm, CrowdStrike, confirmed that a defective software update was the cause of the Global IT outage and was in the process of rolling back the update. Microsoft estimated that the impact began around 19:00 UTC on the 18th of July. Several airports, banks, hospitals, and media outlets soon began experiencing outages across the world and widespread reports of technical difficulties and Blue Screen of Death (BSOD) across platforms came barreling in.
A Blue Screen of Death Source from CNBC TV18
The Blue Screen of Death - also known as black screen errors or STOP code errors – will appear when a critical issue forces Windows to shut down or restart unexpectedly. The notice occurs when the system is experiencing errors in driver software or hardware issues. The BSOD signifies a complete system failure at the Windows kernel level caused by issues with Windows drivers or hardware - rather than an application crash. Users might see a notification such as, "Windows has been shut down to prevent damage to your computer." Once the Blue Screen of Death shows up, workers and users cannot access the system. This is essentially the reality for multiple organizations around the world. While this is still an unfolding story, we can still try to get into some of the effects that the global IT outage has had across the globe so far.
Impact of the Global IT Outage
The impact of this global IT outage is still gaining victims and millions of people are being affected by the ripple effects of the incident as companies struggle to come back online. Already, the CrowdStrike incident is being hailed as one of the largest IT outages in history with an impact that could span billions of people. According to BBC, Microsoft estimated that CrowdStrike update outage affected 8.5 million Windows devices. Let’s look at some of the industries affected so far:
Broadcasting
Major broadcasting and news agencies across the world began experiencing technical difficulties as the CrowdStrike outage took root. NBC News, MSNBC, and Sky News are among the main agencies facing broadcast interruptions and having to use backup options to bring coverage to viewers. Australian broadcasters such as Sky News Australia, ABC, SBS, Channel 7, and Channel 9 also reported issues. Since then, most of these broadcasters have been able to come back online.
Airports
Airports took the biggest brunt of the CrowdStrike Microsoft outage where over 1000 flights have been cancelled globally. Major delays began on Friday morning as booking systems crashed. Planes were also forced to stay grounded as their computer systems were affected. In the US, United, American, Delta, and Allegiant Airlines had all been grounded. A comprehensive list of the affected airports thus far as reported by Sky News includes:
- Heathrow - Flights are "operational" but there are delays
- Luton - Using manual systems for check-in services
- London Gatwick - Warning of delays
- Manchester - Check-in taking longer for some airlines
- Edinburgh airport - Longer wait times due to outage
- Stansted Airport - Check-in services being done manually
- Liverpool Airport - Airlines affected
- Birmingham Airport - Some delays at check-in
- Belfast Airport - Whiteboard being used to provide flight details
- Berlin Brandenburg Airport in Germany
- All Spanish Airports
- Amsterdam Airport Schiphol
- Budapest Airport
- Sydney Airport in Australia
- Singapore's Changi Airport
- Hong Kong Airport
- Narita Airport in Japan
- Prague Airport in Czechia
- Melbourne Airport in Australia
- Zurich Airport in Switzerland
The Paris Olympics organizers said that the outage affected their computer systems and the arrival of some delegations, uniforms, and accreditations were delayed, however, the outages did not affect ticketing or the torch relay. Australian airports saw massive lines and stranded passengers as online check-in services and self-service booths were disabled – even though flights were still operating. Swissport is one of the world's biggest ground handling services for airports, check-ins, and baggage and has also been impacted.
Sourced from BBC
Many people took to social media to discuss the impact of the global outage, complain about the massive delays and share the global IT outage updates. In India, Hong Kong, and Thailand, many airlines were forced to manually check in passengers and physically write out boarding passes when the system went down.
Sourced from X
Railways in the UK and the US Metro Rail were also delayed by the outage for a time. The Dutch carrier KLM said it had been “forced to suspend most” of its operations.
Healthcare
NHS England said that the "majority of GP surgeries" are affected and that users have been unable to book appointments or access patient records. Services reportedly affected included IT service desks, transport booking systems, radiology reporting, the NHS App, and more. Emergency services affected by the global IT outage also involved 911 lines in Alaska, Arizona, Indiana, Minnesota, New Hampshire, and Ohio. Many independent pharmacies were also unable to access prescriptions and medicine deliveries while some hospitals in northern Germany canceled all elective surgery scheduled for Friday – however, emergency care remained unaffected. Mass General Brigham is also canceling all previously scheduled non-urgent surgeries, procedures, and medical visits.
Banks
In South Africa, both Capitec Bank and ABSA Bank reported nationwide service disruptions, however, services were restored soon after. The New Zealand banks ASB and Kiwibank said their services were down as well. Other banks affected included Nationwide, Santander, Lloyds, HSBC, NatWest, Bendigo Bank, and Adelaide Bank.
Logistics
Shipping was disrupted by the outage too when a major container hub in the Baltic port of Poland reported issues as well.
How CrowdStrike and Microsoft Reacted?
Following the initial disruptions, Microsoft noted that it was aware of an issue affecting Windows devices due to an update from a “third-party software platform.” The tech giant promised that a resolution would be “forthcoming. Microsoft 365 posted on social media that the company was “working on rerouting the impacted traffic to alternate systems to alleviate impact” and that they were “observing a positive trend in service availability.”
George Kurtz, the CEO of CrowdStrike, assured the public that the organization was “actively working with customers impacted by a defect found in a single content update for Windows hosts," He also confirmed that Mac and Linux hosts were not impacted. He maintained that the incident was not a security incident or cyber-attack. Kurtz reassured that the issue had been identified, isolated, and a fix had been deployed.
Kurtz further referred customers to the support portal for the latest updates and assured the public that the company would continue to provide complete and continuous updates on its website. "We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels,” Kurtz cautioned while stating that his team is “fully mobilized to ensure the security and stability of CrowdStrike customers."
The recording played by the CrowdStrike answering machine relates that the company is aware of the reports of crashes on Microsoft ports related to the CrowdStrike Falcon sensor - one of the company’s products used to block online attacks. Kurtz has apologized for the outage, stating that the company is “deeply sorry” for the impact caused to customers, to travelers, to anyone affected by this – including the company itself. He went on to state that it could be “some time for some systems that won't automatically recover," but that the company "would make sure every customer is fully recovered." Shares of both companies were losing ground in premarket trading on Friday morning. With such a far-reaching impact, the incident has also caught the attention of several other authorities.
How to Fix the CrowdStrike Issues
According to Forbes, some users have resolved the problem by rebooting their computers. CrowdStrike has offered a manual workaround solution for the blue screen error. This workaround involves booting the system into Safe Mode or the Windows Recovery Environment and navigating to the C:\Windows\System32\drivers\CrowdStrike directory. In this directory, users need to delete the file named “C-00000291*.sys.” This process disables CrowdStrike and other third-party drivers from operating, as reported by The Verge.
Microsoft has also provided a recovery tool with two repair options:
- Recover from WinPE – this option produces boot media that will help facilitate the device repair.
- Recover from safe mode – this option produces boot media so impacted devices can boot into safe mode. The user can then login using an account with local admin privileges and run the remediation steps.
Third-Party Opinions on the CrowdStrike Outage
Many governments, cybersecurity agencies, and third parties have offered their statements concerning the global IT outage. German Chancellor, Olaf Scholz, said that German security institutions are working with international partners to resolve an IT outage that has affected air travel, banking, and several companies.
David Seymour, New Zealand’s acting prime minister, also said that officials in the country were “moving at pace to understand the potential impacts,” adding that he had no information indicating it was a cybersecurity threat.
National Cyber Security Coordinator in Australia, Michelle McGuinness, stated that the issue was causing “inconvenience” for the public and businesses. A spokesperson for the U.S. National Security Council further told CNBC that they were “aware of the incident and are looking into the issue and impacts.”
Switzerland’s National Cyber Security Service (NCSC) said it has received “corresponding reports from various companies and critical infrastructures in Switzerland” amid ongoing global system failures that the agency blamed on CrowdStrike.
On Reddit, community users of the CrowdStrike subreddit (r/crowdstrike) shared what is reported to be an advisory from the company issued to customers only that suggests the cause is the CrowdStrike Falcon Sensor. The notice states: "CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. Symptoms include hosts experiencing a bug check\blue screen error related to the Falcon Sensor."
Possible Reasons for the Global IT Outage
The role of a cybersecurity company is to use privileged access to software that will prevent unauthorized intrusion. However, this double-edged sword also means that cybersecurity companies can become a vulnerability for organizations as well – just as CrowdStrike became for Microsoft.
According to an alert sent by CrowdStrike to its clients and reviewed by Reuters, the widely-used "Falcon Sensor" software created by CrowdStrike was the cause of the Microsoft Windows crash and Blue Screen of Death. The CrowdStrike Falcon Sensor is an endpoint security product meant to run in the background and secure the ends of the network from cyber threats. However, a simple faulty update to the software led the endpoints to crash meaning that the issue cannot be fixed remotely and the issue needs to be resolved manually – endpoint by endpoint.
Mr. Kurtz stated that “the system was sent an update and that update had a software bug in it and it caused an issue with the Microsoft operating system," He went on to say that the company identified this very quickly and remediated the issue. CrowdStrike has since issued advice about a temporary workaround for tech companies:
- Boot Windows into Safe Mode or the Windows Recovery Environment (you can do that by holding down the F8 key before the Windows logo flashes on screen)
- Navigate to the C:\Windows\System32\drivers\Crowdstrike directory
- Locate the file matching the “C-00000291*.sys” file, right click and rename it to “C-00000291*.renamed”
- Boot the host normally.
CrowdStrike Releases Root Cause Analysis of the IT Outage
On August 6th, rowdStrike provided a full analysis of the root cause, offering details. The main issue was a "count mismatch" between the expected input fields of CrowdStrike’s Falcon driver and those provided in a content update. CrowdStrike has committed to improving its update testing procedures and has enlisted the assistance of two separate third-party software security vendors to review its sensor code and release processes.
Security incidents like this have a devastating effect on companies and industries on a massive scale. Cybersecurity platforms are meant to protect and defend systems from compromise and this global IT outage has pointed out that even these fortresses of protection can become a vulnerability. As the CrowdStrike outage unfolds and the consequences become known, it presents a critical moment for users and companies where defenses are laid flat and hackers may take advantage. Several fallout cyber threats are now imminent – especially the potential for ransomware attacks.
Contact Sangfor today for information on enhancing your cloud infrastructure and cybersecurity or visit www.sangfor.com to learn more.
