Supply Chain Security Footguns¶

Mistakes that cause outages or security incidents.

1. Signing Tags Instead of Digests¶

You sign ghcr.io/org/myapp:v1.2.3 by tag. An attacker with registry write access pushes a different image to the same tag. The old signature now points to a digest that no longer matches the tag. Consumers who verify cosign verify ghcr.io/org/myapp:v1.2.3 get a signature that refers to the old image — the new malicious image passes unchecked unless verification is strict about digest matching.

Fix: Always sign by digest, not tag: cosign sign ghcr.io/org/myapp:v1.2.3@sha256:<digest>. In CI, capture the digest from the build step output and sign that specific digest. Deploy with pinned digests rather than mutable tags.

Gotcha: Docker Hub, GHCR, and ECR all allow tag mutation by default. The latest tag is the worst offender — it changes on every push. Even "release" tags like v1.2.3 can be overwritten unless the registry enforces tag immutability. Enable tag immutability in ECR (imageTagMutability: IMMUTABLE) and use OCI referrers for signatures so they follow the digest, not the tag.

2. Admission Controller in Audit Mode Forever¶

You install Kyverno or OPA/Gatekeeper in audit mode (validationFailureAction: Audit) to evaluate policies without blocking. You fix some violations, declare success, and forget to flip it to Enforce. Weeks later, a developer deploys an unsigned image from DockerHub and the admission controller logs a warning — which nobody reads. Supply chain verification is theater.

Fix: Treat audit mode as a temporary ramp — set a deadline (1-2 weeks) to resolve violations and switch to Enforce. Use kubectl get policyreport -A to monitor audit findings daily during the ramp period. Alert on new audit violations via Prometheus metrics exposed by Kyverno or Gatekeeper.

3. SBOM Attached But Never Scanned¶

You generate an SBOM and attach it to the image as a nice-to-have compliance checkbox. No CI job actually scans the SBOM for vulnerabilities, and no Kubernetes admission policy checks for an SBOM attestation. The SBOM is metadata that nobody reads. A critical CVE lands in one of your dependencies and sits undetected in production for months.

Fix: The SBOM is only useful if you scan it. Add grype sbom:./sbom.spdx.json --fail-on high as a required CI gate that blocks the build. Separately, set up a scheduled scan (daily/weekly) against your production image SBOMs to catch newly-published CVEs in already-deployed images.

4. Pinning Dependencies by Name Without Locking the Hash¶

Your Dockerfile uses COPY requirements.txt . and pip install -r requirements.txt where requirements.txt pins package versions (requests==2.31.0) but not hashes. A compromised PyPI account republishes a patched version of requests==2.31.0 with different content but the same version string. Your next build silently installs the malicious package.

Fix: Use hash-pinned dependency files: pip install --require-hashes -r requirements.txt. Generate with pip-compile --generate-hashes. For Go, the go.sum file already provides this. For npm, use npm ci (which requires package-lock.json) rather than npm install. Regularly run pip audit / npm audit / govulncheck against locked files in CI.

CVE: The event-stream npm incident (CVE-2018-16396) — a maintainer transferred ownership of a popular package to an attacker who injected a cryptocurrency-stealing payload in a new version. Hash-pinning wouldn't have helped because the malicious code was in a new, legitimate-looking version. This is why you also need npm audit and regular dependency review, not just hash locking.

5. Third-Party GitHub Actions Pinned to a Mutable Tag¶

Your workflow uses uses: actions/checkout@v4 or uses: some-org/some-action@main. If the upstream repository is compromised or the tag is moved, your next CI run pulls malicious code with full access to your secrets and build environment. Supply chain attacks via GitHub Actions are a documented, exploited attack vector.

Fix: Pin every third-party action to a full commit SHA:

uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1

Use a tool like tj-actions/changed-files audit or Renovate with pinDigests: true to automate pin updates. Review action source code for suspicious curl | sh patterns before first use.

6. Image Registry Credentials in CI Without Rotation¶

Your CI/CD pipeline uses a long-lived personal access token or service account credential to push images to the container registry. The credential is stored as a CI secret, never rotated, and shared across all pipelines. When an engineer leaves, the credential lives on. When the CI system has a breach, the credential is exposed and attackers gain push access to your image registry.

Fix: Use OIDC keyless authentication wherever possible — GitHub Actions can authenticate to ECR, GAR, and GitHub Container Registry without long-lived credentials. For registries that require credentials, use short-lived tokens via workload identity (e.g., AWS IRSA, GCP Workload Identity). If static credentials are unavoidable, rotate them quarterly and use per-repo credentials with minimal permissions (push to specific repos only).

7. Trusting Base Image Tags in Production Builds¶

Your Dockerfile starts with FROM python:3.11-slim. This tag is mutable — the Python project can push a new image to this tag at any time, including security patches or (if the registry is compromised) malicious content. Your builds pull whatever the tag points to at build time, with no verification that it matches what you previously tested.

Fix: Pin base images by digest in production Dockerfiles:

FROM python:3.11-slim@sha256:abc123...

Use tools like docker scan, Trivy, or Grype to verify the base image has no critical CVEs before building. Automate base image digest updates via Renovate or Dependabot so pinning doesn't mean permanently staying on old images.

8. Private Key Checked into Git¶

You generate a cosign key pair for image signing (cosign generate-key-pair) and commit cosign.key to the repo "temporarily" because the CI secret wasn't set up yet. Even after deleting the file in a later commit, the key exists in git history. Anyone with access to the repo — including future employees, compromised CI/CD systems, or leaked repo archives — can extract and use the signing key.

Fix: If a private key is committed to git, treat it as fully compromised: generate a new key pair immediately, revoke trust in the old key, and re-sign all artifacts with the new key. In the repository, git filter-repo --path cosign.key --invert-paths to purge the key from history (and force-push). Prefer keyless signing (OIDC) which eliminates long-lived private keys entirely.

War story: GitHub reports that over 50% of security alerts they issue to repository owners involve leaked secrets in commit history. Their secret scanning service detects tokens for AWS, GCP, Slack, and hundreds of other services — but cosign keys and custom signing material are not always detected. Tools like gitleaks and trufflehog scan for entropy patterns that catch what GitHub's scanner misses.

9. SLSA Level 1 Claimed But Build Is Not Reproducible¶

You generate provenance attestation and claim SLSA Build Level 1 compliance, but your builds are not reproducible — timestamps, random UUIDs, or environment variables are baked into the artifact. Two builds from the same source produce different digests. A consumer cannot independently verify the artifact by re-building from source, making the provenance only as trustworthy as your CI system's integrity.

Fix: SLSA levels are about the trustworthiness of the build process, not just generating an attestation file. For reproducible builds: remove embedded timestamps (SOURCE_DATE_EPOCH), use deterministic file ordering, avoid non-deterministic UUIDs at build time. Tools like rebuilderd can verify Go binary reproducibility. Work toward SLSA Level 2 (hosted build, version-controlled build script) before claiming Level 3 (hardened build, non-falsifiable provenance).

10. Dependency Confusion Attack Surface¶

Your internal packages are published to a private registry under short names (e.g., internal-utils, myorg-logging). Your pip install or npm install configuration falls back to the public registry (PyPI, npmjs) when a package isn't found in the private registry. An attacker registers internal-utils on PyPI with a higher version number — package managers that check public registries by default will install the malicious public package instead.

Fix: Configure package managers to use only your private registry with no public fallback for internal packages. For npm: set registry in .npmrc to your private Artifactory/Nexus, and use --registry flag. For pip: use --index-url (not --extra-index-url) to avoid fallback. Use namespaced package names (e.g., @myorg/utils) that cannot be registered by outsiders on public registries.

War story: Alex Birsan's 2021 dependency confusion research compromised Apple, Microsoft, PayPal, Shopify, Netflix, Tesla, and Uber using this exact technique. He registered public packages matching internal package names and got code execution inside these companies' build systems. The payloads phoned home with hostnames and usernames, proving the attack worked. Microsoft paid him $40,000 through their bug bounty.