Skip to content

Virtualization Footguns

  1. Overcommitting CPU without monitoring contention. You assign 4 vCPUs to 20 VMs on a 32-core host (80 vCPUs total). It works — until all VMs get busy at once. CPU steal time spikes, everything becomes sluggish, and you can't tell which VM is the victim vs the cause.

Fix: Monitor CPU steal time (%steal in top/vmstat) inside guests and CPU ready time on the hypervisor. Keep CPU overcommit below 3:1 for general workloads, 1:1 for latency-sensitive ones. Alert when steal time exceeds 5%.

  1. Ignoring NUMA topology on multi-socket servers. You let the hypervisor auto-place a 64GB VM on a dual-socket server. The VM's memory spans both NUMA nodes. Every cross-node memory access adds 40-100ns, and the database inside the VM gets inconsistent latency that is impossible to explain from inside the guest.

Fix: Pin VMs to a single NUMA node when possible. Use virsh numatune and virsh vcpupin to keep vCPUs and memory local. Check topology with numactl --hardware before sizing VMs.

Under the hood: On a dual-socket server, each CPU has its own memory controller. Accessing "local" memory (attached to the same socket) takes ~80ns. Accessing "remote" memory (other socket) takes ~130ns — a 60% penalty. For a database VM doing millions of memory accesses per second, cross-NUMA traffic can cost 15-25% of total throughput. numastat -c qemu-kvm shows per-NUMA-node allocation for all VMs.

  1. Thin-provisioning all disks and never monitoring actual usage. You create 50 VMs each with 200GB thin-provisioned qcow2 disks on a 2TB storage pool. Total virtual allocation: 10TB. Everything is fine until the guests actually start writing data and the host filesystem fills up, crashing all VMs simultaneously.

Fix: Monitor the storage pool's actual allocation, not just virtual size. Set alerts at 70% actual capacity. Track growth rate so you can add storage proactively. Use qemu-img info and virsh pool-info in monitoring.

  1. Never cleaning up snapshots. You create snapshots before upgrades but forget to delete them afterwards. Weeks later, you have a snapshot chain 15 layers deep. Each disk I/O traverses the chain. Performance degrades steadily. Deleting old snapshots now takes hours because QEMU must merge layers.

Fix: Delete snapshots within hours of a successful change. Monitor snapshot count per VM — alert if any VM has more than 2. Include snapshot cleanup in your change management checklist.

  1. Running without VirtIO drivers. You install a VM and accept the defaults: emulated IDE disk, emulated e1000 NIC. Disk throughput is 30% of what VirtIO delivers. Network throughput is halved. You troubleshoot the application for weeks before realizing the hypervisor emulation layer is the bottleneck.

Fix: Always specify bus=virtio for disks and model=virtio for NICs during virt-install. For Windows guests, load the VirtIO driver ISO during installation. Audit existing VMs with virsh dumpxml | grep bus=.

Debug clue: Inside the guest, lspci | grep -i virtio should show VirtIO devices. If you see "Intel 82545EM" (emulated e1000) or "Intel 82801" (emulated IDE), you're running emulated hardware. Compare disk performance: fio --name=test --rw=randread --bs=4k --runtime=10 typically shows 2-3x IOPS improvement from emulated IDE to VirtIO.

  1. Using virsh destroy as routine shutdown. virsh destroy is the equivalent of yanking the power cord. Using it routinely risks filesystem corruption, database corruption, and lost in-flight writes. "But it's faster than waiting for shutdown" is the rationale that precedes data loss.

Fix: Use virsh shutdown for graceful ACPI shutdown. If the guest doesn't respond, investigate the guest agent or ACPI configuration instead of defaulting to destroy. Reserve virsh destroy for actual emergencies.

Gotcha: virsh shutdown sends an ACPI power button event. If the guest OS doesn't have acpid running (common on minimal installs), the signal is silently ignored and the VM stays running. You wait, nothing happens, and reach for virsh destroy. Fix: install acpid in every guest during provisioning, or use virsh shutdown --mode=agent which requires qemu-guest-agent instead of ACPI.

  1. No console access configured in the guest. The VM loses network connectivity. You try virsh console and get a blank screen because the guest kernel doesn't have a serial console configured. Your only option is VNC, which also doesn't work because the VNC port is firewalled.

Fix: Configure console=ttyS0,115200 on the kernel command line for every VM at build time. Also ensure the VM XML includes <serial> and <console> devices. Test console access during provisioning, not during an incident.

  1. Live migrating VMs with incompatible CPU features. You migrate a VM from a Haswell host to an IvyBridge host. The VM was using AVX2 instructions. It crashes or panics on the destination because the CPU feature isn't available. The application's SIMD-optimized code suddenly hits illegal instruction faults.

Fix: Use cpu mode='custom' with a common baseline model across your cluster, or ensure all hosts in a migration pool have identical CPU models. Test migration both directions before relying on it. VMware EVC and oVirt CPU profiles solve this systematically.

  1. Forgetting VM autostart after host maintenance. You reboot the hypervisor for kernel updates. Half the VMs don't come back because autostart was never set. You spend 30 minutes figuring out which VMs should be running and manually starting them.

Fix: Set virsh autostart <vm> as part of your VM provisioning workflow. Audit with virsh list --all --autostart regularly. Include a post-reboot validation step in your host maintenance runbook.

  1. Running nested virtualization in production. You use nested VMs for your CI pipeline because it was easy to set up. Builds are slow, flaky, and consume 30% more CPU than they should. The inner hypervisor lacks hardware acceleration for some features, causing subtle test failures.

    Fix: Use containers for CI workloads whenever possible. If you must test VM-based workflows, dedicate bare-metal hosts for nested virt and accept the overhead. Never run latency-sensitive or customer-facing workloads in nested VMs.