Container Base Images — Street Ops

Practical patterns for choosing, building, and maintaining container images.

---

Quick-Start: Which Base Image?

Go binary          → scratch or distroless/static
Rust binary        → scratch or distroless/static
Python app         → python:3.12-slim (or Alpine if no C extensions)
Node.js app        → node:20-alpine (or node:20-slim if native modules)
Java app           → eclipse-temurin:21-jre-alpine (or distroless/java)
.NET app           → mcr.microsoft.com/dotnet/aspnet:8.0-alpine
Ruby app           → ruby:3.3-slim
General tooling    → debian:bookworm-slim
Enterprise/RHEL    → ubi9/ubi-minimal
Maximum security   → Chainguard or distroless

---

Under the hood: "Distroless" images from Google contain only the application runtime (libc, libssl, ca-certificates) with no shell, package manager, or OS utilities. This dramatically reduces the attack surface — fewer binaries means fewer CVEs and no shell for an attacker to drop into. The tradeoff: you cannot exec into the container for debugging.

Dockerfile Best Practices

Layer Caching

# GOOD: dependencies change less often than code
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .                     # code changes don't bust pip cache

# BAD: every code change reinstalls dependencies
COPY . .
RUN pip install -r requirements.txt

Minimize Layer Size

# GOOD: single RUN, clean up in same layer
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
      curl \
      ca-certificates && \
    rm -rf /var/lib/apt/lists/*

# BAD: cleanup in separate layer (original data still in image)
RUN apt-get update
RUN apt-get install -y curl
RUN rm -rf /var/lib/apt/lists/*    # too late — previous layers keep the data

Non-Root User

# Debian/Ubuntu
RUN useradd -r -s /sbin/nologin -u 1000 appuser
USER appuser

# Alpine
RUN adduser -D -u 1000 appuser
USER appuser

# Distroless — already runs as nonroot
FROM gcr.io/distroless/static-debian12:nonroot

.dockerignore

# .dockerignore
.git
.github
.venv
__pycache__
*.pyc
node_modules
.env
*.md
tests/
docs/

---

Debugging Containers Without a Shell

# Distroless or scratch — can't exec in
# Options:

# 1. Use debug variant
# gcr.io/distroless/base-debian12:debug  (has busybox shell)

# 2. Ephemeral debug container (k8s)
kubectl debug -it pod/myapp --image=busybox -- sh
kubectl debug -it pod/myapp --image=nicolaka/netshoot -- bash

# 3. Copy files out
kubectl cp pod/myapp:/app/logs/error.log ./error.log

# 4. Port-forward and test from outside
kubectl port-forward pod/myapp 8080:8080
curl localhost:8080/health

# 5. Add debug stage in Dockerfile
FROM gcr.io/distroless/static-debian12 AS prod
COPY --from=build /server /server
CMD ["/server"]

FROM debian:bookworm-slim AS debug
COPY --from=build /server /server
RUN apt-get update && apt-get install -y curl strace
CMD ["/server"]

# Build specific stage: docker build --target debug -t myapp:debug .

---

Image Size Audit

# Check image size
docker images myapp

# Dive deep into layers
docker history myapp:latest
# Or use `dive` tool for interactive layer analysis:
# dive myapp:latest

# Find what's eating space
docker run --rm myapp:latest du -sh /* 2>/dev/null | sort -rh | head

# Compare bases
docker pull alpine:3.20 && docker images alpine
docker pull debian:bookworm-slim && docker images debian
docker pull ubuntu:22.04 && docker images ubuntu

---

Alpine Troubleshooting

# DNS issues — Alpine's musl resolv behaves differently
# Symptom: "Temporary failure in name resolution" intermittently

# Fix 1: Add ndots option
RUN echo "options ndots:1" >> /etc/resolv.conf

# Fix 2: Install glibc NSS compatibility
RUN apk add --no-cache gcompat

# Build issues — missing headers
RUN apk add --no-cache \
    build-base \      # gcc, make, etc.
    python3-dev \     # Python headers
    musl-dev \        # musl headers
    linux-headers \   # kernel headers
    libffi-dev \      # FFI (needed by many Python packages)
    openssl-dev       # OpenSSL (needed by cryptography, etc.)

# Timezone issues
RUN apk add --no-cache tzdata
ENV TZ=America/New_York

---

Regular Rebuild Pipeline

# .github/workflows/rebuild.yml
name: Weekly rebuild
on:
  schedule:
    - cron: '0 4 * * 1'   # Every Monday 4am

jobs:
  rebuild:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build --no-cache -t myapp:latest .
      - run: trivy image --exit-code 1 --severity HIGH,CRITICAL myapp:latest
      - run: docker push myapp:latest

Rebuild weekly even without code changes — picks up base image security patches.

Gotcha: docker build --no-cache rebuilds all layers, but if your FROM image tag (e.g., python:3.12-slim) was already pulled and cached locally, Docker uses the local copy. To ensure you get the latest base image patches, run docker pull python:3.12-slim before building, or use --pull with buildx.

---

Multi-Arch Builds

# Build for both AMD64 and ARM64
docker buildx create --use
docker buildx build --platform linux/amd64,linux/arm64 \
  -t myapp:latest --push .

Base image arch support: - Alpine: amd64, arm64, arm/v7, arm/v6, etc. - Debian-slim: amd64, arm64, arm/v7 - Distroless: amd64, arm64 - UBI: amd64, arm64, s390x, ppc64le

Image Optimization Operations

Find What's Taking Space

# Check the biggest layers
docker history myapp:latest --format "{{.Size}}\t{{.CreatedBy}}" | sort -rh | head -5

# Interactive layer explorer
dive myapp:latest

# Check for common bloat sources
docker run --rm myapp:latest du -sh /usr/lib /usr/share /var/cache /root/.cache 2>/dev/null | sort -rh

# List all files in image
docker export $(docker create myapp:latest) | tar -tv | sort -k5 -rh | head -20

Fix Layer Cache Invalidation

# Check .dockerignore exists and excludes:
# .git, __pycache__, *.pyc, node_modules, tests/, docs/, .env*

# Copy dependency files first, then install, then copy source:
# COPY requirements.txt .
# RUN pip install -r requirements.txt
# COPY . .

BuildKit Cache Management

# Check BuildKit cache usage
docker buildx du
docker buildx prune --filter until=48h

# Build with CI cache
docker buildx build \
  --cache-from type=registry,ref=ghcr.io/org/myapp:buildcache \
  --cache-to type=registry,ref=ghcr.io/org/myapp:buildcache,mode=max \
  -t myapp:latest .

Default trap: Alpine uses musl libc instead of glibc. Most pre-compiled Python wheels on PyPI are built against glibc. On Alpine, pip install falls back to building from source, which requires build-base, python3-dev, and library headers — turning a 5-second install into a 5-minute compilation that bloats your image. If your Python app has C extensions (numpy, psycopg2, cryptography), prefer python:3.12-slim over Alpine.

Image Scanning Operations

Scan and Gate in CI

# Quick scan
trivy image myapp:latest

# CI gate — fail on critical/high
trivy image --exit-code 1 --severity CRITICAL,HIGH --ignore-unfixed myapp:latest

# JSON output for parsing
trivy image --format json myapp:latest | jq '.Results[].Vulnerabilities[] | select(.Severity == "CRITICAL")'

# Generate SBOM
syft packages myapp:latest -o cyclonedx-json > sbom.cdx.json

# Scan from SBOM (no Docker daemon needed)
grype sbom:sbom.cdx.json --fail-on critical

Compare Vulnerability Surfaces

# Before/after base image upgrade
trivy image python:3.11-slim --format json > old.json
trivy image python:3.12-slim --format json > new.json
diff <(jq -r '.Results[].Vulnerabilities[]?.VulnerabilityID' old.json | sort) \
     <(jq -r '.Results[].Vulnerabilities[]?.VulnerabilityID' new.json | sort)

---

Quick Reference

• Deep Dive: Docker Image Internals

Pages that link here

• Anti-Primer: Container Base Images
• Container Images
• Docker Image Internals