--- tags: - devops - l1 - flashcard-deck - runbook-craft --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [Runbook Craft](../../../../library/portal/topics.md) | **Domain:** DevOps & Tooling id category difficulty tags question answer source_path runbook-craft/aef4453592a4 runbook-craft easy runbook, anatomy What are the five sections every effective runbook must have? "(1) Trigger — what alert/symptom activates this runbook, (2) Diagnose — commands to run before taking action, (3) Act — the fix with decision trees for multiple scenarios, (4) Verify — confirm the fix worked, (5) Escalate — when and how to call for help.\n\nRemember: ""A runbook is a recipe for operations."" It turns tribal knowledge into repeatable procedures.\n\nRemember: ""TDAVE"" = Trigger, Diagnose, Act, Verify, Escalate. Every runbook needs all five." training/library/topics/runbook-craft/primer.md runbook-craft/37f8dd7d0b0f runbook-craft easy runbook, metrics, thresholds Why should runbooks include observable thresholds with baselines instead of vague descriptions? "High latency is meaningless without a number. A good runbook says ""Check p99 latency. If > 500ms (normal: 80-120ms), proceed."" Including the normal baseline lets the responder know what ""good"" looks like, enabling faster diagnosis at 3 AM.""\n\nGotcha: Runbooks rot fast — schedule quarterly reviews. A wrong runbook is worse than no runbook.\n\nRemember: ""Runbook rule #1: test it. Rule #2: keep it current.""\n\nExample: Instead of ""check if latency is high,"" write: ""Check p99 latency. If > 500ms (normal: 80-120ms), proceed to step 3.""" training/library/topics/runbook-craft/primer.md runbook-craft/4f096744f953 runbook-craft easy runbook, automation, levels What are the five automation levels for runbooks (L0-L4)? "L0: fully manual prose instructions. L1: copy-paste commands. L2: scripts with parameters. L3: triggered scripts with human approval (e.g., chatbot). L4: fully automated self-healing (no human in the loop). The goal is to move every runbook toward L4 over time, but L1 is infinitely better than nothing.\n\nRemember: ""L0=prose, L1=paste, L2=script, L3=approve, L4=auto."" Every runbook should climb from L0 toward L4 over time." training/library/topics/runbook-craft/primer.md runbook-craft/f52cece7f93f runbook-craft medium runbook, decision-tree Why do complex incident runbooks need decision trees instead of linear steps? "Complex incidents have multiple possible causes requiring different remediation paths. Decision trees provide branching logic — ""IF recent deploy caused it: rollback. IF dependency is down: page owning team and enable circuit breaker. IF pods are OOM-killed: increase memory limit."" This prevents the on-call engineer from guessing which path to follow.\n\nAnalogy: A decision tree runbook is like a flowchart — each branch leads to a different fix. Linear runbooks assume only one cause." training/library/topics/runbook-craft/primer.md runbook-craft/e67c46e00d77 runbook-craft medium runbook, testing, game-day What is a game day and how does it validate a runbook? "A scheduled event where a failure is injected into a system and a responder (deliberately chosen as someone unfamiliar with the service) follows the runbook to resolve it. Success criteria: resolution without asking for help, within time limit, all commands work as documented, and escalation path is clear.\n\nName origin: ""Game day"" comes from sports — a scheduled event where you test readiness under realistic conditions." training/library/topics/runbook-craft/primer.md runbook-craft/a58e093c9eca runbook-craft medium runbook, escalation Under what five conditions should a runbook direct the on-call engineer to escalate? "(1) Diagnosis doesn't match any known scenario. (2) Fix didn't work after one attempt. (3) Multiple services are affected. (4) Data integrity may be compromised. (5) You've been working on it for 15 minutes without progress.\n\nRemember: ""Good runbook = copy-paste commands + expected output."" If the operator has to think about syntax, the runbook failed.\n\nRemember: ""15-minute rule: escalate if no progress in 15 minutes."" Better to escalate early than to extend an outage." training/library/topics/runbook-craft/primer.md runbook-craft/a73b79de5ded runbook-craft medium runbook, review, rot What are the five triggers that should prompt a runbook review? "(1) After every incident — update with lessons learned. (2) After every deploy — verify commands still work. (3) Monthly — owner reviews for accuracy. (4) Quarterly — full team walkthrough of critical runbooks. (5) New team member onboarding — have them follow runbooks and report gaps.\n\nGotcha: The most common trigger is missing: ""after a production incident."" Post-incident runbook review should be mandatory." training/library/topics/runbook-craft/primer.md runbook-craft/64c746e92862 runbook-craft hard runbook, bad-runbooks Why are bad runbooks worse than no runbooks at all? Bad runbooks give false confidence, contain outdated commands, and send engineers down wrong paths during incidents. An engineer following a stale runbook trusts it is correct, wasting precious incident time on invalid steps. No runbook at least signals uncertainty, prompting the engineer to investigate from first principles or escalate sooner.\n\nWar story: An engineer followed a stale runbook that said to restart the primary database. The runbook was written before the HA setup, and the restart caused a failover cascade. training/library/topics/runbook-craft/primer.md runbook-craft/02a687e54127 runbook-craft hard runbook, chaos, testing How does chaos engineering relate to runbook validation, and what tool can automate this? Chaos engineering provides automated failure injection that validates runbooks continuously, not just during scheduled game days. Tools like LitmusChaos define ChaosEngine experiments that target specific applications (e.g., pod-delete for api-gateway), automatically inject failures, and verify that the system recovers as documented in the runbook.\n\nName origin: Chaos engineering was pioneered by Netflix\'s Chaos Monkey (2011) — randomly terminating production instances to ensure resilience. training/library/topics/runbook-craft/primer.md runbook-craft/dff5becc4575 runbook-craft hard runbook, l2-script What makes an L2 runbook script better than an L1 copy-paste command, and what should it include? An L2 script is parameterized, reusable, and includes validation. It should accept the deployment name and namespace as parameters, provide a usage message if arguments are missing, echo what it is doing for operator awareness, run the remediation command, and verify the result (e.g., kubectl rollout status with a timeout). This eliminates copy-paste errors and provides guardrails.\n\nExample: A good L2 script starts with `#!/bin/bash\nset -euo pipefail` and includes usage(), parameter validation, confirmation prompts, and verification steps. training/library/topics/runbook-craft/primer.md --- ## Wiki Navigation ### Related Content - [Runbook Craft](../../../../library/topics/runbook-craft/index.md) (Topic Pack, L1) — Runbook Craft