CrowdStrike Outage Highlights the Vulnerabilities of Traditional IT Setups
On July 19, 2024, a major IT outage caused by a faulty CrowdStrike update led to widespread disruptions, affecting around 8.5 million Windows-based systems worldwide. The update error triggered a system crash known colloquially as the “Blue Screen of Death” (BSOD). The impact was severe. Numerous sectors, including airline, healthcare, and finance, suffered substantial delays and business interruptions. The fallout from this incident highlights the vulnerabilities of traditional IT setups and the need for more resilient solutions.
Image source: https://www.usatoday.com/
In this article, we explore how Virtual Desktop Infrastructure (VDI) solutions, such as Sangfor VDI, mitigate the impact of such outages. With VDI, businesses achieve rapid recovery, ensuring a swift return to normal operations in minutes rather than hours.
First, let’s recap the root cause of the CrowdStrike issue to understand how VDI provides a more resilient approach to business continuity and disaster recovery.
Root Cause of the CrowdStrike Outage
The recent CrowdStrike outage was caused by a faulty update to the Falcon Sensor software. Specifically, it was a logic error in the update to Channel File 291, which is part of the Falcon platform’s behavioral protection mechanisms. This file controls how the sensor evaluates named pipe execution on Windows systems to detect malicious activities.
Critically, the Falcon Sensor operates at the kernel level and uses a boot-start driver, meaning that it must be loaded to start the OS. As a result, the faulty update led to a system failure rather than just an application failure, which is what caused a BSOD and prevented systems from booting up.
How Sangfor VDI Would Have Mitigated the Failure
1. Backup and Recovery
When a system error occurs, a common solution is to recover from backups. However, this approach is often impractical with traditional PCs due to several limitations:
- Time Consumption and System Unavailability: The biggest drawback of PCs is the need to manually create clones of the system disk. This process can take 10 minutes to several hours, during which the system is offline. As a result, few users proactively perform these backups.
- External Storage Issues: External storage solutions require additional equipment. Using NAS for backup can consume a significant amount of internal bandwidth, potentially causing network congestion.
- Cloud Backup Limitations: Cloud backup can help, but it generally supports file and data backups rather than full system backups and, therefore, is not useful for system failures.
- System Reboot: The system often needs to be rebooted into a recovery environment to restore the OS from a backup. However, in the CrowdStrike incident, this was problematic since the system entered a blue screen upon booting.
Here is how Sangfor VDI addresses these limitations:
- Scheduled Backup and Snapshot: Sangfor VDI automatically backs up virtual desktops through scheduled file backups and snapshots. Snapshots capture the state of a virtual desktop at a specific point in time, ensuring fast, lightweight backups with minimal system impact. This allows for more frequent backups and reduces data loss risk.
- Separation of Backup Files: In VDI, backup files are stored separately from the virtual machine (VM) running the desktop. This means the backup files remain safe and accessible even if the VM encounters critical errors, such as a blue screen of death.
- No Reboot Rollback: Sangfor VDI allows admins and users to revert to a previous snapshot without rebooting the VM. This process avoids the traditional boot sequence, which would be more efficient in incidents like the CrowdStrike update, where booting the machine was not possible.
- Centralized Management: Virtual desktops in a VDI environment are managed centrally. Admins can apply updates, rollbacks, or configurations from a central console, eliminating the need to be physically present at individual devices.
By leveraging these backup capabilities, Sangfor VDI enables businesses to swiftly recover from unexpected faulty updates and other failures, restoring operations within minutes rather than hours.
2. User Profile Management
What if not all the computers in your organization were installed with the new CrowdStrike update? While these systems could take over business operations, they may lack the necessary data and settings for a smooth transition. Preparing these systems and restoring data from backups (e.g., external storage or cloud) can be time-consuming and delay business recovery. Not to mention that there might not be any backup files available.
Sangfor VDI to the Rescue Again
Sangfor VDI addresses the above challenges with its User Profile Management (UPM) technology. Here’s how it works:
- Separation of Data: UPM technology ensures that system data, user configurations, and user files are stored separately from the VM running the virtual desktop.
- Rapid Assignment of VMs: In case of a system crash like the CrowdStrike incident, admins can swiftly assign a new VM to affected users. The user can then load their system data, personal files, and configurations as if they were working on their original virtual desktop.
- Automatic Data Retrieval: The process is fast because the new VM automatically retrieves the data. Admins only need to provide users with access permissions and credentials.
Sangfor VDI’s UMP approach ensures that businesses can maintain operations with minimal disruption, providing a quick and seamless transition for users affected by system failures or updates.
3. Out-of-Band Management
So far, we have learned that VDI allows admins to centrally and remotely manage virtual desktops. However, not all VDI solutions would allow this in the CrowdStrike outage. This depends on the management protocol.
Many VDI solutions use in-band management protocols, where the client connects directly to the VM. This requires the VM’s network to be operational. If the VM’s network is down or unresponsive, the IT team cannot provide timely remote intervention to support the VM. This dependency complicates addressing issues like the CrowdStrike incident even when using VDI, as in-band protocols would not allow remote access to the VMs until the network was restored.
Additionally, users cannot see the boot process with in-band protocols. This means if the VM is slow to boot or fails to start, users are unaware of the issue and may keep restarting to solve the problem.
Sangfor VDI’s Out-of-Band (OOB) Management
Sangfor VDI uses an out-of-band (OOB) management protocol, allowing the client to connect to the hypervisor rather than directly to the VM. Even if the VM experiences a blue screen of death or other network issues, the IT team can remotely access and troubleshoot the VM before it receives an IP address.
This would have been particularly beneficial during the recent CrowdStrike incident. Admins would still have been able to access and fix VMs even if their network was down. This capability is vital for maintaining business continuity and minimizing downtime during critical system failures. Moreover, with OOB protocols, users can see the standard boot process. In the case of the CrowdStrike issue, they would see the BSOD and immediately know the problem.
Lessons Learned
The recent CrowdStrike outage shines a light on the vulnerabilities inherent in traditional IT setups and the need for more resilient solutions. Sangfor VDI offers a robust alternative. It offers seamless backup and recovery capabilities, flexible user profile management, and efficient out-of-band management. With Sangfor VDI, you recover swiftly from disruptions, minimize downtime, and maintain operational continuity. Investing in such advanced virtual desktop infrastructure is a strategic move to safeguard your business against future incidents and enhance overall IT resilience.
Visit the Sangfor VDI webpage to learn more or reach out to us with your inquires.
