Skip to content

The Intern and the DROP TABLE

Category: The Close Call Domains: database-ops, security Read time: ~5 min

Setting the Scene

It was Jake's second week. He was eager, sharp, and had just been handed a Jira ticket to clean up orphaned rows in a development database. His mentor had given him a SQL script and said, "Run this against dev, verify the counts, and close the ticket." Simple enough.

What nobody realized was that Jake's database client, DBeaver, had two connection profiles saved: dev-mysql-rw and prod-mysql-rw. They were right next to each other in the sidebar. The icons were the same color.

What Happened

Jake opened the script. It was straightforward — a few SELECT statements for verification, then a DELETE FROM orphaned_sessions WHERE created_at < '2024-01-01', and at the bottom, a DROP TABLE orphaned_sessions_backup. The backup table was 400GB of data nobody needed anymore. Or so the ticket said.

He connected to what he thought was dev. The SELECT counts looked plausible — dev had been recently refreshed from a prod snapshot, so the numbers were similar. He ran the DELETE. It completed in 1.3 seconds. That should have been his first clue — dev had 50K rows, but prod had 12 million.

He scrolled down to the DROP TABLE statement. He highlighted it. He right-clicked and selected "Execute Statement."

A dialog box appeared: "This connection requires multi-factor authentication for DDL operations. Please approve the push notification on your registered device."

Jake didn't have MFA enrolled. He didn't even have a registered device. Because Jake was an intern, and interns didn't have production DDL permissions.

He called his mentor over. "Hey, I'm getting this weird MFA popup on dev." His mentor leaned over, looked at the connection profile in the bottom status bar, and went pale. "Jake. That's prod."

The Moment of Truth

The MFA gate on production DDL operations stopped the DROP TABLE cold. The DELETE had run, but only against 847 rows that matched the date filter — the WHERE clause saved us from a full table wipe. We restored those 847 rows from the point-in-time replica in 20 minutes. Total customer impact: zero.

The Aftermath

We color-coded every database connection: green for dev, yellow for staging, red for prod. We enforced connection naming conventions in all approved SQL clients. Production read-write access was removed from all individual accounts — you now go through a bastion with session recording. Jake got a coffee mug that said "I almost DROP TABLE'd prod and all I got was this mug." He's a senior engineer now.

The Lessons

  1. Principle of least privilege is not optional: Interns — and honestly most engineers — should never have production write access by default. Access should be requested, justified, time-boxed, and audited.
  2. Production access controls save lives: That MFA gate cost us two sprints to implement. It paid for itself in Jake's second week. Defense in depth means the control works even when the human doesn't.
  3. Make environments visually distinct: If your prod and dev database connections look identical in the client UI, someone will mix them up. It's not a matter of if.

What I'd Do Differently

I'd enforce connection profiles via a managed configuration file distributed by IT, not manually created by each engineer. I'd also add a PROMPT_PROD=true environment variable that makes the SQL client show a full-screen red banner on connection.

The Quote

"The best access control is the one that fires when someone doesn't even know they need it."

Cross-References