Pattern: Wrong Terminal Tab¶
ID: FP-046 Family: Human Error Amplifier Frequency: Very Common Blast Radius: Single Service to Multi-Service Detection Difficulty: Obvious (but irreversible)
The Shape¶
An engineer has two terminal tabs (or tmux panes): one connected to production, one to staging. They type a destructive command (DROP DATABASE, kubectl delete, truncate table) in the tab they believe is staging. The command executes in production. The tab titles, shell prompts, or context indicators were not sufficiently distinctive, or the engineer was tired and moving fast. The command is instantaneous and often irreversible.
How You'll See It¶
In Databases¶
-- Production terminal (engineer thinks it's staging)
DROP DATABASE orders_db;
-- Query OK, 0 rows affected (0.12 sec)
-- All orders data gone
In Kubernetes¶
# Production context (engineer thinks it's staging)
kubectl delete deployment payment-service
# deployment.apps "payment-service" deleted
kubectl config current-context would have shown production-cluster, but it wasn't
checked before the command.
In Linux/Infrastructure¶
# Production server (engineer thinks it's staging)
rm -rf /var/lib/mysql/ # intended for staging database cleanup
db-prod-01, but the engineer was context-switching
between an incident on staging and routine work.
The Tell¶
Destructive command executed on the wrong environment. The engineer had multiple terminal sessions open to different environments. Post-execution:
hostnameorkubectl config current-contextconfirms the wrong environment. The command was correct for the intended environment; the environment was wrong.
Common Misdiagnosis¶
| Looks Like | But Actually | How to Tell the Difference |
|---|---|---|
| Application bug | Human error in wrong environment | Command was correct; environment was wrong |
| Intentional change gone wrong | Unintentional context switch | Engineer did not intend to target this environment |
The Fix (Generic)¶
- Immediate: Restore from backup (FP-025 — if tested); attempt point-in-time recovery (FP-027).
- Short-term: Configure distinctive shell prompts:
PS1='\[\e[31m\]PRODUCTION\[\e[0m\] $ '(red for production); usekubectlcontext aliases that are visually distinct. - Long-term: Require confirmation prompts for destructive commands on production (
alias drop_prod_db="echo 'TYPE YES TO CONFIRM:' && read x && [[ $x == YES ]] && drop_database"— obviously needs real implementation); use separate terminal profiles (Terminal.app background color per environment); implement RBAC that limits destructive operations on production.
Real-World Examples¶
- Example 1: DBA had staging and production MySQL connections in adjacent terminal tabs. Ran
DROP TABLE orderson production. 2.3 million orders deleted. 4-hour restoration from backup. - Example 2: Engineer kubectl deleted a CronJob on production thinking it was staging. The CronJob ran nightly reconciliation; the next morning, 14 hours of transactions were not reconciled. Took 6 hours to identify and manually reconcile.
War Story¶
Two terminals:
ssh db-stagingandssh db-prod. I was switching between them quickly during an incident. I needed to truncate a test table on staging. I executedTRUNCATE TABLE user_sessions. 2.3 million production sessions gone. Every user logged out. The prompt saiddb-prod-01in white text on a black terminal. The staging prompt saiddb-staging-01in white text on a black terminal. The only difference was "prod" vs "staging" in the hostname. I now run production terminals with a red background. You cannot look at a red terminal and not think "production." Zero wrong-tab incidents in the 3 years since.
Cross-References¶
- Topic Packs: database-ops, incident-command
- Footguns: database-ops/footguns.md — "Running DROP DATABASE on wrong connection"
- Related Patterns: FP-025 (untested backup — the recovery path), FP-048 (device name confusion — same wrong-target pattern in storage)