Dell PowerEdge Footguns¶
Hardware mistakes that cause outages, data loss, and 3 AM pages. Most of these are preventable with checklists but devastating when skipped.
1. Using Shared iDRAC NIC Mode Instead of Dedicated¶
You configure iDRAC to share a NIC with the host OS to save a cable and a switch port. The server kernel panics. The shared NIC goes down. You cannot reach iDRAC to diagnose the problem or reboot the server. You drive to the datacenter at 2 AM. The one time you needed out-of-band access most, you do not have it.
Fix: Always use the dedicated iDRAC NIC port on a separate management VLAN. One cable and one switch port per server is cheap insurance. Verify iDRAC is reachable before putting a server into production. Test it by shutting down the host OS and confirming the iDRAC web interface still responds.
2. Leaving Default iDRAC Credentials (root/calvin)¶
The default iDRAC 9 password is root / calvin. Every Dell technician, every contractor, and every search engine result knows this. A scanner on your management VLAN finds 200 iDRACs with default credentials. An attacker now has full hardware control — virtual console, power cycling, virtual media boot from a malicious ISO, and firmware flashing.
Fix: Change iDRAC credentials during initial provisioning, before the server touches production traffic. Use RACADM or Redfish in your automation:
Store credentials in a secrets manager. Rotate them on a schedule. Audit with: racadm get iDRAC.Users.2.UserName across the fleet.
3. Running Firmware Updates During RAID Rebuild¶
A RAID array is rebuilding after a disk replacement. You decide to "save time" by updating the PERC controller firmware while the rebuild is in progress. The firmware update requires a controller reset. The rebuild is interrupted. The array may not resume correctly. In the worst case, the array is marked foreign or the rebuild restarts from zero, extending the vulnerability window by hours.
Fix: Never update PERC firmware during a RAID rebuild. Check rebuild status first:
Wait for the rebuild to complete (100%), verify the array is Optimal, then schedule the firmware update. This also applies to BIOS updates that require a reboot — the rebuild survives a clean OS reboot but not a controller reset.
4. RAID 5 on Large Drives Without Understanding Rebuild Risk¶
You build a RAID 5 array with 6x 8TB HDDs. One disk fails. During the 20+ hour rebuild, every remaining disk is read from start to finish. The probability of an unrecoverable read error (URE) on consumer-grade drives is approximately 1 in 10^14 bits — which is roughly 12TB of reads. With five 8TB drives being read (40TB total), the probability of a URE during rebuild is significant. A URE during RAID 5 rebuild means data loss.
Fix: Use RAID 6 (tolerates two simultaneous failures) or RAID 10 (faster rebuild, better write performance) for arrays with drives larger than 2TB. RAID 5 is acceptable only for small drives (< 1TB) or non-critical data. For software-defined storage (Ceph), use HBA passthrough mode and let Ceph handle redundancy with configurable replication.
Under the hood: The math: enterprise drives spec a URE rate of 1 in 10^15 bits (1 in ~125TB). Consumer drives are 1 in 10^14 (1 in ~12.5TB). With five 8TB drives in a RAID 5 rebuild, you read ~40TB. Against a consumer drive's URE spec, the probability of hitting a URE during rebuild is roughly 40/12.5 = ~96%. Even enterprise drives at 40/125 = ~27% chance of a URE during rebuild. This is why Dell's own PERC documentation recommends RAID 6 for drives over 2TB.
5. Not Monitoring PERC Cache Battery / Supercap Health¶
The PERC controller uses write-back caching to dramatically improve write performance. This requires a battery or supercap to protect the cache contents during a power loss. When the battery dies or the supercap fails, the controller silently falls back to write-through mode. Write performance drops 5-10x. Your database suddenly has 10x write latency. Nobody connects the PERC cache alarm (which fired a week ago and was ignored) to the database slowdown.
Fix: Monitor PERC cache power status as a first-class alert:
Alert when cache power state is not "Optimal". Replace the battery/supercap proactively — they degrade over 2-4 years. Include this check in your pre-deployment health script.
6. Pulling the Wrong Drive in a RAID Array¶
A disk LED blinks amber indicating failure. You walk to the rack and pull a drive from the wrong slot — or worse, the wrong server (they all look identical from the front). A healthy RAID array instantly becomes degraded or offline. If you pull two drives from a RAID 5 array, data is lost.
Fix: Always confirm the exact slot before pulling:
# Blink the LED on the failed drive
perccli /c0/e0/s2 start locate
# Walk to the rack — the blinking drive is the one to replace
# After replacement:
perccli /c0/e0/s2 stop locate
Verify the server identity via the service tag pull-tab on the front panel. Label drive slots during initial rack-and-stack. Never pull a drive on assumption alone.
7. Skipping Firmware Baseline Before OS Install¶
You install the OS immediately after racking a new server. The factory firmware is months old. The BIOS has a known bug that causes intermittent freezes under heavy memory load. The PERC firmware has a bug that corrupts data during RAID rebuild. You discover this six months later when a disk fails and the rebuild corrupts data.
Fix: Always update firmware to a tested baseline before installing the OS. Dell publishes tested bundles via the Dell Repository Manager or Dell System Update (DSU):
# Install DSU on the server
yum install dell-system-update
# Run the update (downloads and applies all available firmware)
dsu --non-interactive
# Or use a pre-built ISO: Dell Server Update Utility (SUU)
Maintain a firmware baseline document for your fleet. Test new firmware on a canary server before rolling out to production.
8. Not Verifying Dual PSU Redundancy After Cabling¶
Both PSUs are installed but both are connected to the same PDU (power distribution unit). When that PDU trips or is taken offline for maintenance, the server loses all power. The second PSU provides zero redundancy because it draws from the same source.
Fix: Connect PSU A to PDU A (A-feed) and PSU B to PDU B (B-feed), sourced from separate circuits. Verify with iDRAC:
Both should show "Presence Detected" and "OK". During datacenter audits, trace cables physically from each PSU to separate PDUs. Label cables: "PSU-A → PDU-A" and "PSU-B → PDU-B".
9. Ignoring ECC Memory Errors (Correctable Errors)¶
The server logs show correctable ECC memory errors on DIMM A3. The server continues running — correctable errors are silently fixed by the memory controller. You ignore them. The error rate increases over weeks. Eventually, a correctable error becomes an uncorrectable error (UE). The kernel panics. The database crashes with corrupted in-memory state. Recovery takes hours.
Fix: Correctable ECC errors are the early warning system for failing DIMMs. Monitor them as a trending metric:
# Check for memory errors
edac-util -s
# Or from iDRAC:
racadm get system.memory
# Or lifecycle log:
racadm lclog view | grep -i 'memory\|dimm\|ecc'
Alert when correctable error rate exceeds a threshold (e.g., >10/day on a single DIMM). Schedule DIMM replacement during the next maintenance window. Do not wait for the uncorrectable error.
10. Configuring HBA Mode When RAID Is Needed (or Vice Versa)¶
You set the PERC controller to HBA passthrough mode because you read it was "better." You install the OS. Later, you try to create a RAID 1 mirror for the boot volume — it fails because the controller is in HBA mode and cannot create virtual disks. Switching modes requires re-initialization of the controller, which destroys all existing disk data.
Fix: Decide the storage architecture before OS install. Use RAID mode for traditional workloads (OS boot volumes, database storage). Use HBA mode only for software-defined storage (Ceph, ZFS, HDFS) where the application manages redundancy. Switching between modes is destructive — it is effectively a full server rebuild.
11. Running BIOS in Non-Performance Mode¶
The server ships with the BIOS power profile set to "Balanced" or "DAPC" (Dell Active Power Controller). This dynamically reduces CPU frequency and transitions cores to lower power states to save electricity. Your latency-sensitive application shows intermittent spikes as cores wake from deep sleep states (C-states). P99 latency jumps from 2ms to 50ms unpredictably.
Fix: For latency-sensitive workloads (databases, trading systems, real-time applications), set the BIOS to "Performance" mode:
racadm set BIOS.SysProfileSettings.SysProfile PerfOptimized
racadm jobqueue create BIOS.Setup.1-1 -r pwrcycle
This disables deep C-states and keeps cores at maximum frequency. Power consumption increases by 20-40%. For general compute (web servers, batch processing), "Balanced" is usually fine.
12. Forgetting to Test iDRAC Access Before Going to Production¶
The server is racked, OS is installed, applications are running. Three months later, a kernel panic. You try to access iDRAC — it is unreachable. The iDRAC IP was never configured on the management VLAN, or the IP conflicts with another device, or the iDRAC NIC cable was never plugged in. You drive to the datacenter.
Fix: Include iDRAC connectivity verification in your provisioning checklist. After racking and cabling, before OS install: 1. Ping the iDRAC IP from the management network 2. Log into the web UI and verify virtual console works 3. Test RACADM SSH access 4. Verify credentials are not default 5. Confirm the lifecycle log is recording events
Document this as a hard gate — no server enters production without confirmed iDRAC access.