Skip to content

Pattern: Wrong Terminal Tab

ID: FP-046 Family: Human Error Amplifier Frequency: Very Common Blast Radius: Single Service to Multi-Service Detection Difficulty: Obvious (but irreversible)

The Shape

An engineer has two terminal tabs (or tmux panes): one connected to production, one to staging. They type a destructive command (DROP DATABASE, kubectl delete, truncate table) in the tab they believe is staging. The command executes in production. The tab titles, shell prompts, or context indicators were not sufficiently distinctive, or the engineer was tired and moving fast. The command is instantaneous and often irreversible.

How You'll See It

In Databases

-- Production terminal (engineer thinks it's staging)
DROP DATABASE orders_db;
-- Query OK, 0 rows affected (0.12 sec)
-- All orders data gone
The only pre-execution signal was the shell prompt hostname or database connection string. Under cognitive load (incident, late night), these are easily overlooked.

In Kubernetes

# Production context (engineer thinks it's staging)
kubectl delete deployment payment-service
# deployment.apps "payment-service" deleted
kubectl config current-context would have shown production-cluster, but it wasn't checked before the command.

In Linux/Infrastructure

# Production server (engineer thinks it's staging)
rm -rf /var/lib/mysql/   # intended for staging database cleanup
The hostname in the prompt was db-prod-01, but the engineer was context-switching between an incident on staging and routine work.

The Tell

Destructive command executed on the wrong environment. The engineer had multiple terminal sessions open to different environments. Post-execution: hostname or kubectl config current-context confirms the wrong environment. The command was correct for the intended environment; the environment was wrong.

Common Misdiagnosis

Looks Like But Actually How to Tell the Difference
Application bug Human error in wrong environment Command was correct; environment was wrong
Intentional change gone wrong Unintentional context switch Engineer did not intend to target this environment

The Fix (Generic)

  1. Immediate: Restore from backup (FP-025 — if tested); attempt point-in-time recovery (FP-027).
  2. Short-term: Configure distinctive shell prompts: PS1='\[\e[31m\]PRODUCTION\[\e[0m\] $ ' (red for production); use kubectl context aliases that are visually distinct.
  3. Long-term: Require confirmation prompts for destructive commands on production (alias drop_prod_db="echo 'TYPE YES TO CONFIRM:' && read x && [[ $x == YES ]] && drop_database" — obviously needs real implementation); use separate terminal profiles (Terminal.app background color per environment); implement RBAC that limits destructive operations on production.

Real-World Examples

  • Example 1: DBA had staging and production MySQL connections in adjacent terminal tabs. Ran DROP TABLE orders on production. 2.3 million orders deleted. 4-hour restoration from backup.
  • Example 2: Engineer kubectl deleted a CronJob on production thinking it was staging. The CronJob ran nightly reconciliation; the next morning, 14 hours of transactions were not reconciled. Took 6 hours to identify and manually reconcile.

War Story

Two terminals: ssh db-staging and ssh db-prod. I was switching between them quickly during an incident. I needed to truncate a test table on staging. I executed TRUNCATE TABLE user_sessions. 2.3 million production sessions gone. Every user logged out. The prompt said db-prod-01 in white text on a black terminal. The staging prompt said db-staging-01 in white text on a black terminal. The only difference was "prod" vs "staging" in the hostname. I now run production terminals with a red background. You cannot look at a red terminal and not think "production." Zero wrong-tab incidents in the 3 years since.

Cross-References

  • Topic Packs: database-ops, incident-command
  • Footguns: database-ops/footguns.md — "Running DROP DATABASE on wrong connection"
  • Related Patterns: FP-025 (untested backup — the recovery path), FP-048 (device name confusion — same wrong-target pattern in storage)