Context

Thousands of Windows machines are experiencing a Blue Screen of Death (BSOD) issue at boot on 19th of July 2024, impacting banks, airlines, TV broadcasters, supermarkets, and many more businesses worldwide. A faulty update from cybersecurity provider CrowdStrike is knocking affected PCs and servers offline, forcing them into a recovery boot loop so machines can’t start properly. The issue is not being caused by Microsoft but by third-party CrowdStrike software that’s widely used by many businesses worldwide for managing the security of Windows PCs and servers.

https://www.theverge.com/2024/7/19/24201717/windows-bsod-crowdstrike-outage-issue

https://www.ndtv.com/world-news/windows-systems-restarting-throwing-blue-screen-of-death-due-to-crowdstrike-error-6138820

https://www.linkedin.com/embeds/publishingEmbed.html?articleId=8652465308060248179

Through this research I want to help all the engineers doing extraordinary work right in these moments in order to restore their servers and basically to restore the business in which they are operating.

Prerequisites

Before running the script, ensure you have the following:

pip install azure-identity azure-mgmt-compute azure-mgmt-network azure-mgmt-resource

az login

Technical explanations

This script automates the process of identifying unhealthy Azure VMs, creating a snapshot of their OS disks, cloning these disks, attaching them to a temporary VM AS DataDisk, running a cleanup script on the cloned disks, and then swapping the cleaned disks back into the original VMs AS OS Disk.

*OS disk could also be swapped directly from VM to another but that means the temporary VM might not start(not tested yet). That’s why this trick with the OS disk->Data Disk->OS Disk means a lot in this case.

For now I build an orchestration script which supports the repair of a Windows VM which was broken by the CrowdStrike update by using a quickfix. You can send the specific unhealthy VM name or just let the script get all the unhealthy instances. The script will interact with the cloud resources through the Azure SDK.

Here’s a detailed explanation of each part of the script:

Authentication Setup

from azure.identity import ClientSecretCredential
from azure.mgmt.compute import ComputeManagementClient
from azure.mgmt.network import NetworkManagementClient
from azure.mgmt.resource import ResourceManagementClient

# Define constants
SUBSCRIPTION_ID = ''
RESOURCE_GROUP_NAME = ''
TEMP_VM_NAME = 'temp-vm-name'
LOCATION = 'westeurope'
TEMP_VM_SIZE = 'Standard_B1ls'
ADMIN_USERNAME = ''
ADMIN_PASSWORD = '' 
TENANT_ID = '' 
CLIENT_ID = ''
CLIENT_SECRET = '' 

# Authenticate using the ClientSecretCredential
credential = ClientSecretCredential(tenant_id=TENANT_ID, client_id=CLIENT_ID, client_secret=CLIENT_SECRET)

# Create management clients
compute_client = ComputeManagementClient(credential, SUBSCRIPTION_ID)
network_client = NetworkManagementClient(credential, SUBSCRIPTION_ID)
resource_client = ResourceManagementClient(credential, SUBSCRIPTION_ID)

Authentication Method:

  • ClientSecretCredential: This method uses a service principal for authentication. You need to have an application in Azure Entra ID, assign it necessary roles, and provide its client ID, client secret, and tenant ID. (picked option)
  • Other Authentication Methods alternatives:
  • DefaultAzureCredential: Automatically handles the most common authentication methods (service principal, managed identity, Azure CLI, etc.).
  • InteractiveBrowserCredential: Prompts for user authentication via a web browser.
  • AzureCliCredential: Uses credentials from the Azure CLI.

Creating Management Clients:

  1. ComputeManagementClient: Used for managing Azure virtual machines.
  2. NetworkManagementClient: Used for managing network resources like virtual networks and network interfaces.
  3. ResourceManagementClient: Used for managing Azure resource groups.

The Azure Microsoft EntraID application

The secret creation:

The role mappings for the application:

Virtual Network and Subnet Creation

def create_virtual_network():
    vnet_params = {
        'location': LOCATION,
        'address_space': {
            'address_prefixes': ['10.0.0.0/16']
        }
    }
    try:
        vnet = network_client.virtual_networks.get(RESOURCE_GROUP_NAME, 'virtNetw')
        print(f"Virtual network 'virtNetw' already exists.")
    except Exception as e:
        if "ResourceNotFound" in str(e):
            vnet = network_client.virtual_networks.begin_create_or_update(RESOURCE_GROUP_NAME, 'virtNetw', vnet_params).result()
            print(f"Virtual network 'virtNetw' created.")
        else:
            raise e
    return vnet

def create_subnet(vnet_name):
    subnet_params = {
        'address_prefix': '10.0.1.0/24'
    }
    try:
        subnet = network_client.subnets.get(RESOURCE_GROUP_NAME, vnet_name, 'default')
        print(f"Subnet 'default' already exists in virtual network '{vnet_name}'.")
    except Exception as e:
        if "ResourceNotFound" in str(e):
            subnet = network_client.subnets.begin_create_or_update(RESOURCE_GROUP_NAME, vnet_name, 'default', subnet_params).result()
            print(f"Subnet 'default' created in virtual network '{vnet_name}'.")
        else:
            raise e
    return subnet

Explanation:

  • create_virtual_network: This function checks if a virtual network exists. If not, it creates one with a specified address space.
  • create_subnet: This function checks if a subnet exists within the virtual network. If not, it creates one.

Network Interface and VM Creation

def create_network_interface(vnet_name, subnet_name):
    subnet = network_client.subnets.get(RESOURCE_GROUP_NAME, vnet_name, subnet_name)
    nic_name = f'bsod433_{str(uuid.uuid4())[:8]}'
    nic_params = {
        'location': LOCATION,
        'ip_configurations': [{
            'name': 'ipConfig1',
            'subnet': {
                'id': subnet.id
            },
            'private_ip_allocation_method': 'Dynamic'
        }]
    }
    nic = network_client.network_interfaces.begin_create_or_update(RESOURCE_GROUP_NAME, nic_name, nic_params).result()
    return nic.id

def create_virtual_machine(nic_id):
    vm_params = {
        'location': LOCATION,
        'hardware_profile': {
            'vm_size': 'Standard_B1ls'
        },
        'os_profile': {
            'computer_name': TEMP_VM_NAME,
            'admin_username': '',
            'admin_password': '',
            'windows_configuration': {
                'provision_vm_agent': True
            }
        },
        'network_profile': {
            'network_interfaces': [{
                'id': nic_id
            }]
        },
        'storage_profile': {
            'image_reference': {
                'publisher': 'MicrosoftWindowsDesktop',
                'offer': 'Windows-10',
                'sku': 'win10-21h2-pro',
                'version': 'latest'
            },
            'os_disk': {
                'caching': 'ReadWrite',
                'managed_disk': {
                    'storage_account_type': 'Standard_LRS'
                },
                'name': TEMP_VM_NAME,
                'create_option': 'FromImage'
            }
        },
        'zones': ['1']
    }
    return compute_client.virtual_machines.begin_create_or_update(RESOURCE_GROUP_NAME, TEMP_VM_NAME, vm_params).result()

Explanation:

  • create_network_interface: This function creates a network interface in a specified subnet.
  • create_virtual_machine: This function creates a new Windows 10 Pro virtual machine using the specified network interface.

Disk and Snapshot Management

def get_os_disk_id(vm_name):
    vm = compute_client.virtual_machines.get(RESOURCE_GROUP_NAME, vm_name)
    return vm.storage_profile.os_disk.managed_disk.id

def get_vm_zone(vm_name):
    vm = compute_client.virtual_machines.get(RESOURCE_GROUP_NAME, vm_name)
    return vm.zones[0] if vm.zones else None

def create_snapshot(disk_id):
    snapshot_name = f'snapshot-{str(uuid.uuid4())[:8]}'
    snapshot_params = {
        'location': LOCATION,
        'creation_data': {
            'create_option': 'Copy',
            'source_resource_id': disk_id
        }
    }
    return compute_client.snapshots.begin_create_or_update(RESOURCE_GROUP_NAME, snapshot_name, snapshot_params).result().id

def create_disk_from_snapshot(snapshot_id, zone):
    disk_name = f'disk-{str(uuid.uuid4())[:8]}'
    disk_params = {
        'location': LOCATION,
        'creation_data': {
            'create_option': 'Copy',
            'source_resource_id': snapshot_id
        },
        'zones': [zone]
    }
    return compute_client.disks.begin_create_or_update(RESOURCE_GROUP_NAME, disk_name, disk_params).result().id

Explanation:

  • get_os_disk_id: Retrieves the OS disk ID of a given VM.
  • get_vm_zone: Retrieves the availability zone of a given VM.
  • create_snapshot: Creates a snapshot of a specified disk.
  • create_disk_from_snapshot: Creates a new disk from a specified snapshot in a specified zone.

Temporary VM Management

def deallocate_vm(vm_name):
    compute_client.virtual_machines.begin_deallocate(RESOURCE_GROUP_NAME, vm_name).result()

def update_os_disk(vm_name, new_disk_id):
    vm = compute_client.virtual_machines.get(RESOURCE_GROUP_NAME, vm_name)
    vm.storage_profile.os_disk.managed_disk.id = new_disk_id
    vm.storage_profile.os_disk.name = new_disk_id.split('/')[-1]
    compute_client.virtual_machines.begin_create_or_update(RESOURCE_GROUP_NAME, vm_name, vm).result()

def start_vm(vm_name):
    compute_client.virtual_machines.begin_start(RESOURCE_GROUP_NAME, vm_name).result()

Explanation:

  • deallocate_vm: Deallocates a given VM.
  • update_os_disk: Updates the OS disk of a given VM to a new disk.
  • start_vm: Starts a given VM.

PowerShell Command Execution

def execute_powershell_command(vm_name):
    command = r'''
    $filePath = "C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys"
    if (Test-Path $filePath) {
        $fileInfo = Get-Item $filePath
        $fileTimestamp = $fileInfo.LastWriteTimeUTC
        $goodTimestamp = Get-Date "2021-05-27T00:00:00Z"
        $problematicTimestamp = Get-Date "2021-04-09T00:00:00Z"
        if ($fileTimestamp -lt $problematicTimestamp) {
            Remove-Item -Path $filePath -Force
            Write-Output "Problematic file removed."
        } elseif ($fileTimestamp -ge $goodTimestamp) {
            Write-Output "File is good, no action taken."
        } else {
            Write-Output "File does not match any known timestamps, no action taken."
        }
    } else {
        Write-Output "File not found."
    }
    '''
    run_command_parameters = {
        'command_id': 'RunPowerShellScript',
        'script': [command]
    }
    poller = compute_client.virtual_machines.begin_run_command(RESOURCE_GROUP_NAME, vm_name, run_command_parameters)
    result = poller.result()
    print(result.value[0].message)

Explanation:

  • execute_powershell_command: Executes a PowerShell command on a given VM to check and remove a problematic CrowdStrike driver file if it exists.

Identifying and Processing Unhealthy VMs

def get_unhealthy_vms():
    vm_list = compute_client.virtual_machines.list(RESOURCE_GROUP_NAME)
    unhealthy_vms = []
    for vm in vm_list:
        if vm.storage_profile.os_disk.os_type == 'Windows':
            vm_instance_view = compute_client.virtual_machines.instance_view(RESOURCE_GROUP_NAME, vm.name)
            if vm_instance_view.statuses[1].code != 'PowerState/running':
                unhealthy_vms.append(vm.name)
    return unhealthy_vms

Explanation:

  • get_unhealthy_vms: Identifies VMs that are not in a running state and returns their names.

At this moment we consider unhealthy just the stopped instances but a more complex logic can be developed, like checking after a specific boot message.

Main VM Processing Function

def process_vm(vm_name, temp_vm_created):
    print(f"Processing VM: {vm_name}")

    os_disk_id = get_os_disk_id(vm_name)
    vm_zone = get_vm_zone(vm_name)

    print(f"Creating snapshot of OS disk: {os_disk_id}")
    snapshot_id = create_snapshot(os_disk_id)

    print(f"Creating disk from snapshot in zone {vm_zone}")
    cloned_disk_id = create_disk_from_snapshot(snapshot_id, vm_zone)

    print(f"Deallocating VM: {vm_name}")
    deallocate_vm(vm_name)

    if not temp_vm_created:
        if not temp_vm_exists():
            print("Creating temporary VM...")
            create_virtual_network()
            create_subnet('virtNetw')
            nic_id = create_network_interface('virtNetw', 'default')
            create_virtual_machine(nic_id)
            print("Temporary VM created.")
        else:
            print("Temporary VM already exists.")
        temp_vm_created = True

    print(f"Attaching cloned OS disk to temporary VM: {TEMP_VM_NAME}")
    vm = compute_client.virtual_machines.get(RESOURCE_GROUP_NAME, TEMP_VM_NAME)
    data_disk = {
        'lun': 1,
        'name': cloned_disk_id.split('/')[-1],
        'create_option': 'Attach',
        'managed_disk': {'id': cloned_disk_id}
    }
    vm.storage_profile.data_disks.append(data_disk)
    compute_client.virtual_machines.begin_create_or_update(RESOURCE_GROUP_NAME, TEMP_VM_NAME, vm).result()

    execute_powershell_command(TEMP_VM_NAME)

    print(f"Detaching cloned OS disk from temporary VM: {TEMP_VM_NAME}")
    vm.storage_profile.data_disks = [disk for disk in vm.storage_profile.data_disks if disk.managed_disk.id != cloned_disk_id]
    compute_client.virtual_machines.begin_create_or_update(RESOURCE_GROUP_NAME, TEMP_VM_NAME, vm).result()

    print(f"Updating VM: {vm_name} to use the cleaned cloned OS disk")
    update_os_disk(vm_name, cloned_disk_id)

    print(f"Starting VM: {vm_name}")
    start_vm(vm_name)

    return temp_vm_created

Explanation:

  • process_vm: Handles the entire process for a given VM, including creating a snapshot, cloning the disk, attaching it as a DataDisk to a temporary VM, running the cleanup script, detaching the cleaned disk, and swapping it back into the original VM.

Main Function

def main():
    parser = argparse.ArgumentParser(description="Process a single VM or all unhealthy VMs.")
    parser.add_argument('--vm-name', type=str, help="The name of the VM to process")

    args = parser.parse_args()

    temp_vm_created = False

    if args.vm_name:
        temp_vm_created = process_vm(args.vm_name, temp_vm_created)
    else:
        unhealthy_vms = get_unhealthy_vms()
        for vm_name in unhealthy_vms:
            temp_vm_created = process_vm(vm_name, temp_vm_created)

    if temp_vm_created:
        print("Deleting temporary VM...")
        delete_virtual_machine()
        print("Temporary VM deleted.")

    print("Script execution completed.")

if __name__ == "__main__":
    main()

Explanation:

  • main: Parses command-line arguments to determine if a specific VM should be processed or if all unhealthy VMs should be processed. It handles the creation and deletion of the temporary VM as needed.

Running the script

python your_script_name.py --vm-name your-vm-name  # For a specific VM
python your_script_name.py  # For all unhealthy VMs

*The script will be probably pushed in github and there can be viewed/maintained by everybody interested.

*Later stages or different orchestration methods for this might be developed.

***The above work was fully tested on a specific instance at this moment in Azure cloud ecosystem.

****The script can be also viewed like used for different kind of purposes for when the VM doesn’t boot and some operations need to be made on the OS Disk.