Mental Model: Runbook-Driven Recovery¶
Category: Operational Reasoning Origin: Operations and systems administration practice; formalized in SRE and DevOps culture; the term "runbook" originates from mainframe operations, where operators literally ran through a physical book of procedures One-liner: Every alert should have a runbook — codified, step-by-step response procedures that eliminate improvisation and cognitive load under pressure.
The Model¶
Runbook-driven recovery is the operational philosophy that every known failure mode should have a pre-written response procedure, and that procedure should be the default path of execution during an incident. The alternative — improvised recovery — sounds faster but is systematically worse. Under the stress of an active incident at 3 AM, with customers impacted and a manager pinging you for updates, your cognitive capacity is at its lowest. Improvisation in this state produces mistakes: wrong commands, skipped verification steps, incomplete rollbacks, and missed escalation triggers. A runbook is pre-computed decision-making done at calm, considered time, made available at the moment when calm, considered thinking is hardest.
The core value of a runbook is cognitive offload. A well-written runbook converts a complex, multi-step recovery process into a checklist that can be followed without deep system knowledge. This serves two purposes simultaneously: it reduces the skill required to execute a known recovery (enabling broader on-call rotation and safer delegation), and it frees the experienced engineer's cognitive resources for the parts of the incident that are genuinely novel (deciding whether the standard procedure applies, coordinating communication, handling edge cases).
Runbooks also encode institutional knowledge that would otherwise live exclusively in the heads of one or two engineers. "The database failover requires flushing the replication buffer before promoting the replica — skip this and you corrupt the replica's transaction log" is the kind of knowledge that lives in muscle memory after you have been burned once, but is nowhere in any document. Capturing it in a runbook transfers it from a person to a system.
The feedback loop between incidents and runbooks is where the real value compounds. Every incident that required improvisation is evidence that a runbook is missing or wrong. Every time an engineer deviated from a runbook because the actual system state did not match what the runbook assumed is a signal that the runbook needs updating. When postmortem action items include "update the runbook," that update is a reliability improvement: the next engineer who encounters this scenario will recover faster and with fewer mistakes. Runbooks that are never updated after incidents rot quickly and become dangerous — stale procedures that lead engineers confidently in the wrong direction.
Runbook quality matters more than runbook quantity. A bad runbook is worse than no runbook: it gives engineers false confidence, leads them through incorrect steps, and damages trust in the process. A good runbook is specific (exact commands, not general guidance), verified (actually tested against the real system, not written from memory), current (reviewed after every relevant incident), and contextual (explains why each step exists, not just what to do). The "why" is critical: when an engineer encounters a system state that the runbook does not cover, understanding the reasoning behind each step allows them to adapt intelligently rather than freeze.
Visual¶
ALERT → RUNBOOK → RECOVERY CHAIN
────────────────────────────────────────────────────────────
Alert fires
│
▼
Runbook linked directly from alert annotation
│
▼
┌─────────────────────────────────────────────────┐
│ RUNBOOK: High Disk Usage on /var/log │
│ │
│ Trigger: disk_usage_percent > 85 on /var/log │
│ Owner: @platform-team Severity: P2 │
│ Last validated: 2026-02-10 │
│ │
│ Step 1: Verify alert is accurate │
│ $ df -h /var/log │
│ Expected: usage > 85% │
│ If <85%: alert is stale, silence and page PM │
│ │
│ Step 2: Identify large consumers │
│ $ du -sh /var/log/* | sort -h | tail -20 │
│ │
│ Step 3: Rotate logs if safe │
│ $ sudo logrotate -f /etc/logrotate.conf │
│ WHY: forces rotation without waiting for cron │
│ │
│ Step 4: If still >90% after rotation: │
│ $ sudo journalctl --vacuum-size=500M │
│ WHY: journal may be unbounded; 500M is safe │
│ CAUTION: do not use --vacuum-time on prod, │
│ it removes security audit logs │
│ │
│ Step 5: Escalate if >95% and steps 3-4 failed: │
│ Page @storage-team, attach du output │
│ │
│ Post-incident: file ticket to add log size cap │
└─────────────────────────────────────────────────┘
│
▼
Resolution documented, runbook updated if needed
RUNBOOK QUALITY SPECTRUM
────────────────────────────────────────────────────────────
POOR GOOD
────────────────────────────────────────────────────────
"Check if the database is running" "Run: systemctl status postgresql.service
Expected output: active (running)
If inactive: proceed to Step 4 (failover)"
"Restart the service if needed" "Run: sudo systemctl restart api-gateway.service
Wait 30 seconds, then verify:
curl -sf http://localhost:8080/health
Expected: HTTP 200 with {status: ok}
If not: do NOT restart again — escalate"
"Contact the database team" "Page #database-oncall via PagerDuty
Attach: pg_activity output + last 50 lines of
/var/log/postgresql/postgresql-*.log"
When to Reach for This¶
- When writing a new alert: the alert is not complete until the runbook is written and linked in the alert annotation — shipping an alert without a runbook is shipping a pager without instructions
- When an on-call engineer improvised a recovery (successfully or not): the improvised steps should be captured and formalized into a runbook immediately after the incident
- When onboarding a new team member to on-call: runbooks are the training material — an engineer should not carry a pager for a service until they have read and executed (in a test environment) the runbooks for its common failure modes
- When a service is being transferred between teams: runbooks encode the operational knowledge that otherwise walks out the door with the previous owners
- When preparing for a production change that carries rollback risk: write the rollback runbook before executing the change
When NOT to Use This¶
- Do not write runbooks for genuinely novel incidents — you cannot pre-write a procedure for a failure mode you have not seen; in those cases, the OODA loop is the right model, and the postmortem produces the runbook
- Do not let runbooks become a crutch that prevents engineers from developing system understanding; a runbook that is followed without understanding what each step does produces dangerous brittleness — when the system state is even slightly different from what the runbook assumes, the engineer has no basis for adaptation
- Avoid runbook sprawl: too many runbooks that are never updated are worse than a smaller set of well-maintained ones — prune and consolidate regularly
Applied Examples¶
Example 1: Disk full on root — services down¶
A monitoring alert fires at 02:17: "Root filesystem 100% — critical." The on-call engineer is woken from sleep.
Without a runbook: The engineer connects to the server and starts guessing. They look in /var/log, find some large files, delete a few, services come back. But they deleted a security audit log that compliance requires, and they did not identify the underlying cause. The disk fills again three days later. Total recovery: 38 minutes, with a compliance violation and a recurring incident.
With a runbook: The runbook linked from the alert specifies: (1) never delete from /var/log/audit/ — these are compliance-critical; (2) check /var/log/journal/ first, it is commonly unbounded; (3) run du -sh /var/log/* /tmp/* /home/* to identify the real consumer; (4) the most common cause is application logs not being rotated — run logrotate; (5) after clearing space, file a ticket to add log rotation and a pre-full alert at 80%. Total recovery: 11 minutes, no compliance violation, root cause identified.
Example 2: SELinux denying a service after OS update¶
After a routine OS update, a web application starts failing with permission denied errors. The service logs show AVC denials. The on-call engineer has never seen an SELinux issue before.
With a runbook: The runbook for "service failing with permission denied after update" includes a branch: "Check /var/log/audit/audit.log for AVC. If present, see SELinux runbook." The SELinux runbook explains: (1) never set SELinux to permissive in production without an incident commander's approval — it disables a security control; (2) use audit2why to interpret the denial; (3) if the denial matches a known pattern, apply the labeled fix; (4) if novel, escalate to security team. The engineer follows the runbook, identifies it as a file context mismatch from the update, runs restorecon -Rv /var/www, service recovers. Total recovery: 18 minutes, SELinux stays enforcing.
The Junior vs Senior Gap¶
| Junior | Senior |
|---|---|
| Treats runbooks as optional documentation written after things have calmed down | Treats runbooks as a prerequisite for shipping an alert — the alert and runbook are deployed together |
| Writes runbooks as narrative descriptions ("you should check the database") | Writes runbooks as executable procedures with exact commands, expected outputs, and branch conditions |
| Never updates runbooks after incidents — "it worked eventually" | After every incident, identifies where the runbook was incomplete or wrong and updates it immediately |
| Has a runbook folder with 30 documents, most last updated two years ago | Maintains a smaller set of high-quality, recently-validated runbooks; obsolete ones are deleted |
| Follows the runbook mechanically even when the system state is clearly different | Understands the "why" behind each step, so they can adapt when the system state diverges from what the runbook assumes |
| Treats deviation from the runbook as failure | Treats deviation from the runbook as a signal that the runbook needs updating; documents the deviation and its outcome |
Connections¶
- Complements: OODA Loop (runbooks pre-load the Decide phase of the OODA loop for known failure patterns — when Orient produces a recognized scenario, the decision is already in the runbook, dramatically shortening loop time)
- Complements: Blameless Postmortem (postmortems frequently produce "update the runbook" action items; runbooks are the systemic fix that prevents the next engineer from being in the same position as the current incident's responder)
- Tensions: Toil vs Automation ROI (a runbook is the correct intermediate step before automation — you must understand and document a process before automating it; but a runbook that has been followed 50 times unchanged is a toil signal and a strong automation candidate)
- Topic Packs: incident-management
- Case Studies: disk-full-root-services-down (the absence of a runbook led to compliance log deletion and a recurring incident), selinux-denying-service (the runbook's explicit "do not disable SELinux enforcement" instruction prevented a security control from being bypassed under pressure)