Skip to content

Anti-Primer: Dagger

Everything that can go wrong, will — and in this story, it does.

The Setup

A DevOps team is building a Dagger pipeline for a critical production service. The pipeline must handle builds, tests, and deployments for 15 microservices. The team is under pressure to ship the pipeline by Friday so developers can start using it next sprint.

The Timeline

Hour 0: No Pipeline Secrets Rotation

Hardcodes deployment credentials in pipeline environment variables and never rotates them. The deadline was looming, and this seemed like the fastest path forward. But the result is leaked credentials from a forked repo give an attacker deploy access to production.

Footgun #1: No Pipeline Secrets Rotation — hardcodes deployment credentials in pipeline environment variables and never rotates them, leading to leaked credentials from a forked repo give an attacker deploy access to production.

Nobody notices yet. The engineer moves on to the next task.

Hour 1: Skipping Tests in Pipeline

Adds a 'skip tests' flag for 'emergency deployments' that becomes the default. Under time pressure, the team chose speed over caution. But the result is a bug that tests would have caught reaches production; 2-hour customer-facing outage.

Footgun #2: Skipping Tests in Pipeline — adds a 'skip tests' flag for 'emergency deployments' that becomes the default, leading to a bug that tests would have caught reaches production; 2-hour customer-facing outage.

The first mistake is still invisible, making the next shortcut feel justified.

Hour 2: No Artifact Pinning

Pipeline pulls dependencies from latest without lockfiles or checksums. Nobody pushed back because the shortcut looked harmless in the moment. But the result is a compromised dependency is pulled during build; malicious code deployed to production.

Footgun #3: No Artifact Pinning — pipeline pulls dependencies from latest without lockfiles or checksums, leading to a compromised dependency is pulled during build; malicious code deployed to production.

Pressure is mounting. The team is behind schedule and cutting more corners.

Hour 3: Shared Pipeline Runner with Secrets

Multiple teams share a CI runner that has access to production credentials. The team had gotten away with similar shortcuts before, so nobody raised a flag. But the result is a developer's test job on the shared runner can read another team's production secrets.

Footgun #4: Shared Pipeline Runner with Secrets — multiple teams share a CI runner that has access to production credentials, leading to a developer's test job on the shared runner can read another team's production secrets.

By hour 3, the compounding failures have reached critical mass. Pages fire. The war room fills up. The team scrambles to understand what went wrong while the system burns.

The Postmortem

Root Cause Chain

# Mistake Consequence Could Have Been Prevented By
1 No Pipeline Secrets Rotation Leaked credentials from a forked repo give an attacker deploy access to production Primer: Use short-lived tokens, OIDC authentication, and rotate secrets on a schedule
2 Skipping Tests in Pipeline A bug that tests would have caught reaches production; 2-hour customer-facing outage Primer: Tests are non-negotiable in the pipeline; no bypass flags
3 No Artifact Pinning A compromised dependency is pulled during build; malicious code deployed to production Primer: Pin all dependencies with lockfiles and verify checksums; use private artifact mirrors
4 Shared Pipeline Runner with Secrets A developer's test job on the shared runner can read another team's production secrets Primer: Isolate runners per environment/team; use ephemeral runners that are destroyed after each job

Damage Report

  • Downtime: Deployment pipeline blocked for 2-8 hours
  • Data loss: None directly, but failed deployments may leave environments in inconsistent states
  • Customer impact: Delayed feature releases; developers blocked from shipping
  • Engineering time to remediate: 8-16 engineer-hours to diagnose, fix, and verify the pipeline
  • Reputation cost: Developer trust in the platform eroded; pressure to bypass safety checks increases

What the Primer Teaches

  • Footgun #1: If the engineer had read the primer, section on no pipeline secrets rotation, they would have learned: Use short-lived tokens, OIDC authentication, and rotate secrets on a schedule.
  • Footgun #2: If the engineer had read the primer, section on skipping tests in pipeline, they would have learned: Tests are non-negotiable in the pipeline; no bypass flags.
  • Footgun #3: If the engineer had read the primer, section on no artifact pinning, they would have learned: Pin all dependencies with lockfiles and verify checksums; use private artifact mirrors.
  • Footgun #4: If the engineer had read the primer, section on shared pipeline runner with secrets, they would have learned: Isolate runners per environment/team; use ephemeral runners that are destroyed after each job.

Cross-References