Skip to content

The Technical Debt Interest Payment

Category: The Hard Lesson Domains: architecture, sre-practices Read time: ~5 min

Setting the Scene

Two years ago, we needed to ship a feature fast. Our PostgreSQL 11 database had a quirk in how it handled jsonb column defaults, and instead of fixing our ORM layer properly, a senior engineer wrote a database trigger that intercepted INSERTs and mutated the JSON payload before writing. "It's a temporary workaround," he said in the PR description. "We'll refactor the ORM layer next quarter."

The trigger worked. The feature shipped. The Jira ticket for the ORM refactor sat in the backlog, reprioritized every sprint, never started.

What Happened

Fast forward two years. That trigger was now load-bearing infrastructure. Fourteen microservices wrote to that table. Three reporting pipelines read from it. The trigger silently reformatted JSON payloads, added default fields, and — this is the part that kept me up at night — enforced a business rule about nested object depth that was documented nowhere except in PL/pgSQL.

Then we needed to upgrade to PostgreSQL 16. Not a want, a need — PG 11 was EOL and our security team had given us a 90-day compliance deadline.

The upgrade itself was straightforward. pg_upgrade handled the data migration fine. But PostgreSQL 16 changed how trigger execution ordering worked with concurrent transactions. Our trigger, which relied on a specific firing order during bulk inserts, started producing corrupted JSON about 3% of the time. Not enough to catch immediately. Enough to poison three days of analytics data before anyone noticed.

We rolled back to PG 11. Then we sat in a room and mapped every dependency on that trigger. The whiteboard looked like a conspiracy theorist's wall — strings connecting fourteen services, three pipelines, two cron jobs nobody remembered creating, and a Zapier integration that the marketing team had set up.

The refactor took 11 weeks. We had to decompose the trigger logic into application-layer middleware, update all fourteen services, rewrite the reporting queries, and validate the output against two years of production data. The original trigger had taken about 2 hours to write.

The Moment of Truth

During the refactor planning meeting, our VP of Engineering asked how a 2-hour workaround became an 11-week project. I pulled up the git blame on the trigger file and showed the original PR. The description read: "Temporary workaround. TODO: refactor ORM layer. Do not build on top of this." Below it were 47 subsequent commits that built on top of it.

The Aftermath

We completed the refactor, upgraded to PG 16, and met the compliance deadline with 8 days to spare. We also introduced a "tech debt register" — a spreadsheet (yes, a spreadsheet, not a Jira board that nobody checks) reviewed monthly by engineering leadership. Every workaround gets a row with an expiration date. When the date passes, it becomes a P1 in the next sprint.

The harder change was cultural. We stopped saying "temporary" in PRs unless there was a linked ticket with an assignee and a due date.

The Lessons

  1. Tech debt compounds like financial debt: A 2-hour shortcut cost 11 weeks because every month, more systems depended on it. The interest rate on tech debt is brutal.
  2. Track your shortcuts: If you write a workaround, log it somewhere with an expiration date. If nobody's tracking it, it's not temporary — it's permanent.
  3. Time-box workarounds: Set a calendar reminder for 30 days after merging a hack. If it's still there, it's a project now, and it needs to be staffed like one.

What I'd Do Differently

I'd add a WORKAROUND label to the PR and a CI check that scans for TODO: refactor or TEMPORARY comments older than 90 days, reporting them in a weekly Slack digest. I'd also require workarounds to include a "removal plan" section in the PR description — not the refactor itself, but the steps needed to remove the workaround when the time comes.

The Quote

"There's nothing more permanent than a temporary workaround that works."

Cross-References