Skip to content

Anti-Primer: Containers Deep Dive

Everything that can go wrong, will — and in this story, it does.

The Setup

A team is optimizing their Containers Deep Dive workflow for a production deployment pipeline. They need to reduce build times and image sizes while maintaining security standards. The deadline is the end of the sprint.

The Timeline

Hour 0: Bloated Base Image

Uses a full OS base image (ubuntu:latest) instead of a minimal image. The deadline was looming, and this seemed like the fastest path forward. But the result is image is 1.2GB with hundreds of unnecessary packages, each a potential vulnerability vector.

Footgun #1: Bloated Base Image — uses a full OS base image (ubuntu:latest) instead of a minimal image, leading to image is 1.2GB with hundreds of unnecessary packages, each a potential vulnerability vector.

Nobody notices yet. The engineer moves on to the next task.

Hour 1: Secrets Baked Into Layers

Copies secrets into the image during build and deletes them in a later layer. Under time pressure, the team chose speed over caution. But the result is secrets are still visible in intermediate layers; anyone with image pull access can extract them.

Footgun #2: Secrets Baked Into Layers — copies secrets into the image during build and deletes them in a later layer, leading to secrets are still visible in intermediate layers; anyone with image pull access can extract them.

The first mistake is still invisible, making the next shortcut feel justified.

Hour 2: No Vulnerability Scanning

Pushes images to the registry without scanning for known CVEs. Nobody pushed back because the shortcut looked harmless in the moment. But the result is a critical CVE in a base package is deployed to production; discovered during a security audit.

Footgun #3: No Vulnerability Scanning — pushes images to the registry without scanning for known CVEs, leading to a critical CVE in a base package is deployed to production; discovered during a security audit.

Pressure is mounting. The team is behind schedule and cutting more corners.

Hour 3: Mutable Tags in Production

Uses mutable tags (latest, stable) for production deployments. The team had gotten away with similar shortcuts before, so nobody raised a flag. But the result is different nodes pull different versions of the same tag; inconsistent behavior across the fleet.

Footgun #4: Mutable Tags in Production — uses mutable tags (latest, stable) for production deployments, leading to different nodes pull different versions of the same tag; inconsistent behavior across the fleet.

By hour 3, the compounding failures have reached critical mass. Pages fire. The war room fills up. The team scrambles to understand what went wrong while the system burns.

The Postmortem

Root Cause Chain

# Mistake Consequence Could Have Been Prevented By
1 Bloated Base Image Image is 1.2GB with hundreds of unnecessary packages, each a potential vulnerability vector Primer: Use minimal base images (distroless, alpine, slim); multi-stage builds
2 Secrets Baked Into Layers Secrets are still visible in intermediate layers; anyone with image pull access can extract them Primer: Use multi-stage builds; pass secrets via build secrets or runtime environment
3 No Vulnerability Scanning A critical CVE in a base package is deployed to production; discovered during a security audit Primer: Scan images in CI before pushing; block deployment of images with critical CVEs
4 Mutable Tags in Production Different nodes pull different versions of the same tag; inconsistent behavior across the fleet Primer: Pin images by digest or use immutable version tags for production

Damage Report

  • Downtime: 1-4 hours of degraded container builds or deployments
  • Data loss: None directly, but vulnerable images may remain in production
  • Customer impact: Delayed releases; potential security exposure from unscanned images
  • Engineering time to remediate: 8-12 engineer-hours for remediation and pipeline repair
  • Reputation cost: Security team flags container practices; additional review gates added

What the Primer Teaches

  • Footgun #1: If the engineer had read the primer, section on bloated base image, they would have learned: Use minimal base images (distroless, alpine, slim); multi-stage builds.
  • Footgun #2: If the engineer had read the primer, section on secrets baked into layers, they would have learned: Use multi-stage builds; pass secrets via build secrets or runtime environment.
  • Footgun #3: If the engineer had read the primer, section on no vulnerability scanning, they would have learned: Scan images in CI before pushing; block deployment of images with critical CVEs.
  • Footgun #4: If the engineer had read the primer, section on mutable tags in production, they would have learned: Pin images by digest or use immutable version tags for production.

Cross-References