Mental Model: Five Whys¶
Category: Debugging & Diagnosis Origin: Sakichi Toyoda, Toyota Production System (1930s–1950s); popularized in DevOps/SRE through the Site Reliability Engineering book and incident retrospective culture One-liner: Ask "why did this happen?" repeatedly — at least five times — until you reach the root cause rather than a surface symptom.
The Model¶
Most incident responses stop too early. An alert fires, an engineer fixes the immediate symptom, the system recovers, and the post-mortem records "disk full — cleaned up log files." A week later the disk is full again. Five Whys is the discipline of not stopping at the symptom. It forces you to keep asking the same question — why did that happen? — until you reach a cause that, if addressed, prevents the entire chain from recurring.
The technique originated in manufacturing, where Sakichi Toyoda observed that defects are almost never the result of a single isolated failure. There is always a chain of causes. The surface symptom is just the point where the chain became visible. Fixing the symptom without tracing the chain leaves the chain intact. Five is not a magic number — it is a heuristic for "more than you're comfortable with." Many chains require three iterations; some require seven. The rule is to keep going until you reach a cause you can actually act on.
The critical discipline is forming each "why" correctly. You are not looking for the next contributing factor — you are asking what specific condition caused the previous answer to be true. The answer to each "why" must be a falsifiable statement about the real world, not a vague attribution ("human error," "it was busy"). "Human error" as an answer means you stopped one step too early — why was the human in a position where an error had that consequence?
Five Whys also reveals organizational and systemic causes that technical analysis alone misses. A chain that starts with "disk full" and ends at "no capacity planning process exists for logging volumes" is pointing at a process gap, not a technical one. This is one of the most valuable things the technique surfaces — the difference between fixing the instance (clean up the disk) and fixing the class (implement log retention policy and monitoring).
The method's limits: it is a causal chain technique, not a causal graph technique. Real incidents often have multiple parallel contributing factors that converge. Five Whys, applied naively, produces one chain. Sophisticated use of the technique involves branching — when you reach a "why" that has two or more independent causes, trace each branch. This is sometimes called "Ishikawa" or "fishbone" analysis.
Visual¶
Five Whys Chain — Linear Form
─────────────────────────────────────────────────────────────────
SYMPTOM: Production API returning 503 errors
Why 1: Why are 503 errors returned?
→ Application pods are crash-looping
Why 2: Why are pods crash-looping?
→ OOM kills: containers exceeding memory limit
Why 3: Why are containers exceeding memory limit?
→ A memory leak was introduced in last deployment
Why 4: Why did the memory leak reach production?
→ No memory usage regression test exists in CI pipeline
Why 5: Why does no such test exist?
→ Memory limits were set once at service creation
and never revisited; no process owns them
ROOT CAUSE: No ownership or review process for memory limits
and no CI gate for memory regression.
ACTION: (1) Fix the leak — immediate
(2) Add memory profiling to CI — short-term
(3) Assign resource limit ownership to teams — systemic
─────────────────────────────────────────────────────────────────
Branching form (when multiple causes exist at one step):
[Symptom]
|
Why 1
|
Why 2
/ \
Why 3a Why 3b ← Two independent causes
| |
Why 4a Why 4b
| |
Root A Root B ← Two fixes needed
flowchart TD
S["503 errors in production"] -->|Why?| W1["Pods crash-looping"]
W1 -->|Why?| W2["OOM kills: memory limit exceeded"]
W2 -->|Why?| W3["Memory leak in last deploy"]
W3 -->|Why?| W4["No memory regression test in CI"]
W4 -->|Why?| W5["No ownership process\nfor resource limits"]
W5 --> FIX1["Fix leak (immediate)"]
W5 --> FIX2["Add memory profiling to CI (short-term)"]
W5 --> FIX3["Assign resource limit ownership (systemic)"]
style S fill:#f55,color:#fff
style W5 fill:#f90,color:#fff
style FIX1 fill:#5a5,color:#fff
style FIX2 fill:#5a5,color:#fff
style FIX3 fill:#5a5,color:#fffWhen to Reach for This¶
- Post-incident retrospectives: any incident where the immediate fix was applied but the underlying cause is unclear
- Recurring incidents: the same symptom appearing more than once is the strongest signal that Five Whys was not applied (or not applied deeply enough) the first time
- When a fix feels like a bandage — cleaning up disk, restarting a service, adding capacity — these are symptoms, not causes
- Process failures: deployment misconfigurations, miscommunications, skipped steps — Five Whys reliably surfaces the process gap
- When the post-mortem team says "human error" — this is never a root cause; it is always a stopping point that requires another "why"
When NOT to Use This¶
- During the live incident: Five Whys is a retrospective tool. Applying it during the heat of an outage consumes attention that should be on mitigation. Document facts during the incident; analyze chains afterward.
- When the causal chain is already clear: if you deployed a bad config and the service broke, you may not need five iterations — two or three suffice. Don't apply the technique mechanically for its own sake.
- As a blame-finding exercise: the technique exposes systemic gaps, not individuals. If the chain terminates with "X person made a mistake," you stopped too early. The follow-up "why" should ask what conditions made that mistake easy to make and hard to catch.
- For novel failures with no established chain: sometimes a failure involves an unknown interaction between systems. Five Whys assumes a traceable chain. If the chain is genuinely unknown, you need investigation first (Differential Diagnosis) before you can apply Five Whys.
Facilitating Five Whys in a Team¶
The technique is most valuable when applied collaboratively during a post-mortem. The facilitator's job is not to provide answers but to prevent the team from stopping too early. Specific facilitation moves:
When the team offers a human action as a cause: Ask "what made that action possible?" or "what would have prevented that action from having this consequence?" These questions pivot from blame to system design.
When the team says "we need better monitoring": This is often a fix masquerading as a root cause. Ask "why did we not know this was happening?" and "why would monitoring have caught it when our existing checks did not?" The root may be that monitoring was never designed for this failure mode — which leads to "why was this failure mode not considered during design?"
When the chain becomes circular: "The system failed because the config was wrong, and the config was wrong because the deploy failed, and the deploy failed because the config was wrong." This usually means you're at the wrong level of abstraction. Step back and re-ask the question at a higher level: what is the source of truth for config? Why can config become inconsistent with what the deploy expects?
When the chain stops at an external system: "The cloud provider had an outage." This is a legitimate terminal point in some cases, but first ask: was there a mitigation we chose not to implement? Did our architecture assume the external system would be available? The root may be an architectural coupling decision that could be reconsidered.
Documentation during the session: Each "why" and its answer should be written in a numbered chain as the team goes. This prevents the team from losing track of the current level and makes the final report clear. The chain is the artifact — it should appear verbatim in the post-mortem.
Distinguishing Root Cause from Proximate Cause¶
A recurring confusion in Five Whys sessions is conflating the proximate cause (the immediate technical trigger) with the root cause (the systemic condition that made the trigger possible and harmful).
Proximate cause: The OOM killer killed the payment service process at 14:23 UTC. Root cause: No process exists to review and revise memory limits when service traffic patterns change, so limits set 18 months ago during initial deployment were never updated despite 4x traffic growth.
The proximate cause is interesting for understanding the incident mechanically. The root cause is what you fix to prevent recurrence. A post-mortem that records only the proximate cause will see the same incident recur with a slightly different technical surface.
A useful test: if you implement the proposed fix, could the same class of incident happen again? If the fix is "raise the memory limit to 2GB," the answer is yes — it will fail again at 2x traffic. If the fix is "implement a quarterly resource limit review tied to traffic baselines," the answer is no — the class of failure is prevented.
Applied Examples¶
Example 1: Systemd service repeatedly flapping — Linux production host¶
Alert: payment-processor.service has restarted 14 times in 2 hours on a single host. Each restart is logged as ExitCode=137 (OOM kill).
Why 1: Why does the service keep restarting? It is being killed by the kernel OOM killer (exit code 137 = SIGKILL from OOM).
Why 2: Why is the OOM killer killing it?
The process is consuming more memory than the systemd MemoryMax limit of 512MB allows, and the system has insufficient free memory to accommodate overages.
Why 3: Why is the process consuming more than 512MB? The service recently started receiving 3x the normal request volume following a capacity redistribution when two other hosts were taken offline for maintenance.
Why 4: Why was this host expected to absorb 3x traffic without resource limit adjustment? The capacity plan assumed uniform distribution across N hosts; when N decreases, per-host load increases, but resource limits are static and not recalculated during maintenance windows.
Why 5: Why do resource limits not account for reduced cluster capacity? No runbook or automation exists to adjust memory limits when cluster capacity changes. The original limits were set during initial provisioning and never revisited.
Root cause: Resource limits are point-in-time values with no process to revise them when the operational environment changes. Fix: (1) Immediately raise the MemoryMax or bring additional hosts online. (2) Create a runbook for maintenance-window capacity planning. (3) Automate resource limit adjustment via autoscaling or Kubernetes resource management.
Example 2: Firmware update causes boot loop — datacenter server¶
A firmware update applied during a scheduled maintenance window results in a server entering a boot loop. It fails to reach the OS.
Why 1: Why is the server in a boot loop? The BIOS POST is failing and triggering an automatic reset, visible in the BMC serial console log.
Why 2: Why is POST failing? The new firmware version is incompatible with the installed RAM configuration (mixed 16GB and 32GB DIMMs from different manufacturers violate a new strict compatibility check added in the firmware).
Why 3: Why was an incompatible firmware version applied to this server? The firmware update script selected the firmware based on server model, not on validated hardware configuration combinations.
Why 4: Why does the update script not validate hardware configuration? The firmware vendor's compatibility matrix was not incorporated into the update tooling; compatibility was assumed by model number alone.
Why 5: Why was compatibility assumed rather than validated? The server fleet was historically homogeneous (identical DIMM configurations). A procurement exception for this rack — using mixed DIMMs to consume remaining inventory — was not communicated to the operations team maintaining the update tooling.
Root cause: A procurement decision created a heterogeneous hardware configuration that was invisible to operational tooling because no process existed to communicate hardware exceptions to operations teams. Fix: (1) Roll back firmware on the affected server. (2) Update the firmware tooling to validate against the vendor compatibility matrix. (3) Create a change management process requiring hardware configuration exceptions to be registered in the CMDB and reviewed by operations.
The Junior vs Senior Gap¶
| Junior | Senior |
|---|---|
| Fixes the immediate symptom (clears disk, restarts service) and closes the ticket | Treats the fix as step one and immediately begins root cause investigation |
| Writes "human error" or "misconfiguration" as the root cause in a post-mortem | Recognizes these as intermediate answers and asks what systemic conditions made the error likely |
| Applies Five Whys linearly and stops at the first plausible cause | Follows all branches when a "why" has multiple independent answers |
| Confuses the root cause with the fix ("we need to monitor disk space") | Separates root cause identification from remediation, ensuring the fix targets the actual root |
| Performs Five Whys during the incident while still fighting the fire | Separates mitigation (incident) from investigation (retrospective) |
| Uses the technique to assign blame ("the engineer ran the wrong command") | Uses the technique to surface systemic gaps ("why could that command cause this damage?") |
Integrating Five Whys into Post-Mortems¶
A blameless post-mortem without Five Whys produces an incident timeline and a list of action items — but action items that address symptoms rather than root causes. The following structure integrates Five Whys directly into the post-mortem format:
Section: Timeline Factual sequence of events from first symptom to resolution. No interpretation yet.
Section: Why did this happen? (Five Whys chain) Start from the user-visible impact and chain downward:
Impact: Users unable to complete checkout for 47 minutes.
Why 1: Why were users unable to complete checkout?
→ Payment API returned 503 for all requests.
Why 2: Why did Payment API return 503?
→ Kubernetes Deployment had 0 available replicas.
Why 3: Why were there 0 available replicas?
→ All pods were in CrashLoopBackOff due to a missing environment variable.
Why 4: Why was the environment variable missing?
→ A required Kubernetes Secret was not created in the production namespace.
Why 5: Why was the Secret not created?
→ The deployment runbook for this service requires manual Secret creation,
but this step was not included in the automated deployment pipeline.
Root cause: Manual Secret creation required by runbook was omitted from the
deployment pipeline, with no automated validation to catch the omission.
Section: What went well? Monitoring detected the incident in 4 minutes. On-call escalation was fast. Rollback procedure was documented.
Section: What could be improved? Directly tied to the root cause: automate Secret creation as part of the deployment pipeline; add a pre-deployment smoke test that verifies all required Secrets exist before routing traffic.
Section: Action items Each action item maps to a specific level in the Five Whys chain, with a clear owner and due date. Action items that only address Why 1 or Why 2 (symptoms) are flagged and escalated — the team must also address the root cause.
This structure prevents the common post-mortem failure where action items are "add monitoring" and "improve runbook" without addressing what made the failure possible in the first place.
Connections¶
- Complements: Differential Diagnosis (use Differential Diagnosis to identify which cause is correct; use Five Whys to trace why that cause exists — they operate in sequence, not in parallel)
- Complements: Bisect (Bisect identifies when the fault was introduced; Five Whys explains why the fault that was introduced has the properties it does — combine them for change-induced regressions)
- Tensions: Correlation vs Causation (Five Whys can construct a plausible-sounding chain that is actually correlation rather than causation — validate each link in the chain with evidence, not just narrative logic)
- Topic Packs: incident-management
- Case Studies: systemd-service-flapping (Five Whys traces the flapping through resource limits back to a process gap in capacity management), firmware-update-boot-loop (Five Whys surfaces a procurement-to-operations communication failure hidden behind a technical symptom)