Skip to content

Supply Chain Security: Trusting Your Dependencies

  • lesson
  • supply-chain-security
  • container-images
  • vulnerability-scanning
  • sbom
  • image-signing
  • admission-controllers
  • ci/cd-security
  • dependency-management ---# Supply Chain Security — Trusting Your Dependencies

Topics: supply chain security, container images, vulnerability scanning, SBOM, image signing, admission controllers, CI/CD security, dependency management Strategy: Adversarial + archaeological Level: L2–L3 (Operations → Advanced) Time: 75–100 minutes Prerequisites: None (everything is explained from scratch)

The Mission

It's 9:47am on a Monday. You're sipping coffee when Slack lights up:

#security-alerts: Trivy scan flagged a CRITICAL CVE in libwebp — present in 43 of our 61 production images. The vulnerable library came from our python:3.11-slim base image, which we haven't rebuilt in 11 weeks.

Your manager asks three questions: 1. Which services are affected? 2. How did this get into production? 3. How do we prevent this from happening again?

You can answer question 1 if you have SBOMs. You can answer question 2 if you understand the supply chain. And question 3 — that's the rest of this lesson.

We'll trace the path from "someone pushed malicious code to npm" all the way to "your Kubernetes cluster refused to run it." Along the way, you'll learn to think like an attacker (to understand the threat), an archaeologist (to understand how we got here), and a builder (to construct defenses that actually work).

Part 1: The Archaeology of Trust

Before we build defenses, we need to understand what broke. The history of supply chain attacks reads like a thriller with escalating stakes.

The Timeline

Year Attack What Happened Impact
2018 event-stream Maintainer handed npm package to a stranger who injected crypto-stealing code 2M weekly downloads compromised
2020 SolarWinds Attackers compromised the Orion build pipeline, injected malware into signed updates 18,000 organizations, including US government agencies
2021 Codecov Bash uploader script modified to exfiltrate CI environment variables Thousands of repos' secrets exposed
2021 ua-parser-js npm package hijacked, crypto miner injected 8M weekly downloads
2021 Log4Shell RCE via log message in Log4j — a transitive dependency most didn't know they had ~93% of enterprise cloud environments
2021 Dependency confusion Alex Birsan registered public packages matching internal names at Apple, Microsoft, etc. 35+ major companies compromised
2024 xz backdoor Attacker spent 2+ years gaining maintainer trust, then injected SSH backdoor Caught by accident — 500ms latency anomaly

Notice the pattern? Each attack is more sophisticated than the last. SolarWinds wasn't some script kiddie — it was a nation-state actor who compromised a build pipeline so that the malware was delivered inside signed, legitimate-looking updates. The xz backdoor was a multi-year social engineering campaign targeting a single burned-out maintainer.

War Story: The event-stream attack in 2018 is the one that should keep every open-source consumer awake at night. The original maintainer, Dominic Tarr, was burned out from maintaining a package he didn't use anymore — for free. When a stranger named "right9ctrl" offered to take over, he handed it off. The new maintainer added a dependency called flatmap-stream that contained obfuscated code targeting the Copay Bitcoin wallet. The malicious code was designed to steal wallet keys only from Copay users, making it hard to detect. The attack was discovered by accident when another developer noticed the suspicious new dependency. The lesson: the open-source supply chain depends on the goodwill of unpaid volunteers, and any one of them can hand the keys to a stranger.

Trivia: The xz backdoor (CVE-2024-3094) was discovered by Andres Freund, a Microsoft engineer working on PostgreSQL. He noticed that SSH logins were taking 500ms longer than expected and decided to investigate. The attacker, operating under the alias "Jia Tan," had spent over two years building trust as an xz contributor — submitting legitimate patches, reviewing code, and slowly gaining commit access. If Freund hadn't been curious about a half-second delay, the backdoor could have shipped in every major Linux distribution and compromised nearly every Linux server on the internet.

Part 2: The Dependency Tree Problem

Here's a question that sounds simple: how many dependencies does your application have?

# Node.js project — count direct + transitive dependencies
ls node_modules/ | wc -l
# 847

# Python project
pip list --format=freeze | wc -l
# 142

# Go project
go list -m all | wc -l
# 213

Most of those are transitive dependencies — things your dependencies depend on, which depend on other things, which depend on other things. You didn't choose them. You probably don't know they exist. But they run with the same permissions as your code.

your-app
├── express (you chose this)
│   ├── body-parser
│   │   ├── bytes
│   │   ├── content-type
│   │   ├── debug
│   │   │   └── ms
│   │   ├── depd
│   │   ├── http-errors
│   │   │   ├── depd
│   │   │   ├── inherits
│   │   │   ├── setprototypeof
│   │   │   ├── statuses
│   │   │   └── toidentifier
│   │   ├── iconv-lite
│   │   │   └── safer-buffer      ← you've never heard of this
│   │   ├── on-finished
│   │   │   └── ee-first
│   │   ├── qs
│   │   │   └── side-channel      ← or this
│   │   ├── raw-body
│   │   │   ├── bytes
│   │   │   ├── http-errors
│   │   │   ├── iconv-lite
│   │   │   └── unpipe
│   │   └── type-is
│   │       ├── media-typer
│   │       └── mime-types
│   │           └── mime-db
│   ├── cookie
│   ...48 more direct deps of express...

Any one of those packages — including the ones you've never heard of — could be compromised. That's the dependency tree problem.

Mental Model: Think of your dependency tree like a building's supply chain. You hired a general contractor (Express), who subcontracted the plumbing (body-parser), who bought pipes from a wholesaler (raw-body), who sourced copper from a mine (unpipe). If anyone in that chain substitutes counterfeit material, your building is compromised — and you won't know until the pipe bursts.

Trivia: Synopsys' annual Open Source Security and Risk Analysis (OSSRA) report consistently finds that 70-80% of commercial application code consists of open-source components. The median commercial application contains over 500 open-source dependencies. You're not building software from scratch — you're assembling it from other people's work.

Flashcard Check #1

Question Answer
What's the difference between a direct and transitive dependency? A direct dependency is one you explicitly declared. A transitive dependency is one pulled in by your dependencies (or their dependencies). You didn't choose it, but it runs with the same privileges.
Why was the event-stream attack hard to detect? The malicious code was in a new transitive dependency (flatmap-stream), was obfuscated, and only activated for a specific target (Copay Bitcoin wallet). It didn't affect most users, so most testing wouldn't catch it.
How was the xz backdoor discovered? Andres Freund noticed SSH logins were 500ms slower than expected and investigated the cause. The discovery was serendipitous — not the result of any security tool or process.

Part 3: Attack Vectors — How They Get You

Supply chain attacks aren't one thing. They're a family of techniques, each targeting a different link in the chain.

Typosquatting

Register a package with a name that looks like a popular one:

lodash     → lodahs, l0dash, lodash-utils
requests   → reqeusts, python-requests, request

A developer makes a typo in pip install or npm install and pulls the malicious package. The package runs an install script that phones home with the host's environment variables.

Dependency Confusion

Your company has internal packages in a private registry:

@myorg/auth-utils     (private, on Artifactory)
@myorg/logging        (private, on Artifactory)

An attacker registers auth-utils on the public npm registry with a higher version number. Your build system, configured to check both registries, installs the public version because it has a higher version number.

War Story: Alex Birsan's 2021 dependency confusion research compromised Apple, Microsoft, PayPal, Shopify, Netflix, Tesla, and Uber using exactly this technique. He registered public packages matching internal package names and got code execution inside these companies' build systems. The payloads phoned home with hostnames and usernames, proving the attack worked. Microsoft paid him $40,000 through their bug bounty program. The whole thing is documented in his blog post "Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies."

Compromised Maintainer Account

An attacker gains access to a maintainer's npm or PyPI account (phishing, credential stuffing, no 2FA) and publishes a new version with malicious code. The package name is legitimate. The version looks like a normal patch. Automated npm install or pip install picks it up on the next build.

The ua-parser-js attack in 2021 used this vector. The package had 8 million weekly downloads. The attacker published versions 0.7.29, 0.8.0, and 1.0.0 with embedded crypto miners and password stealers.

Compromised Build Pipeline

The SolarWinds attack didn't touch the source code at all. Attackers compromised the build server and injected malware during compilation. The source code in version control was clean. Code review wouldn't have caught it. The resulting binary was signed with SolarWinds' legitimate certificate.

This is why SLSA Level 3 requires a hardened build platform — the build environment itself must be tamper-resistant.

The GitHub Actions Supply Chain

Your CI workflows are also a supply chain:

# DANGEROUS — pinned to a mutable tag
- uses: actions/checkout@v4
- uses: some-org/deploy-action@main

# SAFE — pinned to an immutable commit SHA
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
- uses: some-org/deploy-action@a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2  # v2.3.0

If some-org/deploy-action@main is compromised, your next CI run pulls malicious code with full access to your GITHUB_TOKEN, deployment secrets, and build artifacts. Pinning by SHA means a compromised tag can't change what you pull.

Gotcha: GitHub Actions run with whatever permissions your workflow grants. A third-party action with permissions: write-all that you pin to @main is one compromised push away from exfiltrating every secret in your repository. Always pin by SHA, and always apply least-privilege permissions.

Part 4: The Container Image Supply Chain

Your application doesn't just depend on npm packages and Python libraries. It depends on an entire operating system — the base image.

FROM python:3.11-slim

This single line pulls in Debian Bookworm, glibc, OpenSSL, curl, and hundreds of other packages. Each one is a dependency. Each one can have CVEs. And that tag — 3.11-slim — is mutable. The Python project can push a different image to that tag at any time.

How Base Images Become Vulnerable

Day 0:    You build your image on python:3.11-slim (clean, no known CVEs)
Day 14:   CVE-2024-0727 is published for libssl3 (in the base image)
Day 30:   CVE-2024-1234 is published for curl (in the base image)
Day 77:   You still haven't rebuilt. Your image has 2 critical CVEs.
Day 78:   Scanner flags it. Panic.

The fix: pin base images by digest and rebuild regularly.

# Mutable tag — what you get changes over time
FROM python:3.11-slim

# Pinned digest — immutable, always the same image
FROM python:3.11-slim@sha256:2bac43769ace90ebd3ad83e5392295e25dfc58e58543d3ab326c3330b505283c

Under the Hood: A container image digest is a SHA-256 hash of the image's manifest. The manifest lists every layer (also by digest). Changing a single byte in any layer produces a completely different manifest digest. This is content-addressable storage — the name points to the content, not the other way around. Tags are mutable pointers. Digests are immutable facts.

The Base Image Spectrum

Not all base images are equal:

ubuntu:22.04        ~75MB    ~45 known CVEs     glibc, bash, apt
python:3.11-slim    ~52MB    ~12 known CVEs     glibc, bash, apt (trimmed)
alpine:3.20         ~7MB     ~0-2 known CVEs    musl, ash, apk
distroless          ~20MB    ~2 known CVEs      glibc, no shell, no pkg mgr
chainguard/python   ~45MB    ~0 known CVEs      glibc, no shell, SBOM built-in
scratch             0MB      0 known CVEs       literally nothing

Every package you don't ship is a package that can't be exploited.

Trivia: Google created "distroless" images in 2017 after getting tired of patching CVEs in packages their applications never used. A 2023 Sysdig report found that 87% of high/critical CVEs in production containers were in packages that were never loaded at runtime. The industry was patching vulnerabilities in code that never executed.

Part 5: Scanning — Finding What's Already There

You can't fix what you can't see. Image scanning is the first line of defense.

Scanning an Image with Trivy

# Basic scan
trivy image myapp:latest

Output looks like this:

myapp:latest (debian 12.4)

Total: 23 (UNKNOWN: 0, LOW: 8, MEDIUM: 10, HIGH: 4, CRITICAL: 1)

┌──────────────┬────────────────┬──────────┬────────────────┬───────────────┐
│   Library    │ Vulnerability  │ Severity │ Installed Ver  │  Fixed Ver    │
├──────────────┼────────────────┼──────────┼────────────────┼───────────────┤
│ libssl3      │ CVE-2024-0727  │ CRITICAL │ 3.0.11-1       │ 3.0.13-1      │
│ curl         │ CVE-2023-46218 │ HIGH     │ 7.88.1-10      │ 7.88.1-10+d12 │
│ zlib1g       │ CVE-2023-45853 │ HIGH     │ 1:1.2.13       │ (none)        │
│ libgnutls30  │ CVE-2024-0553  │ HIGH     │ 3.7.9-2        │ 3.7.9-2+deb12 │
│ libc6        │ CVE-2023-6246  │ HIGH     │ 2.36-9         │ 2.36-9+deb12  │
└──────────────┴────────────────┴──────────┴────────────────┴───────────────┘

Let's break that down:

Column What It Means
Library The OS package or language dependency with the vulnerability
Vulnerability The CVE identifier — look this up on nvd.nist.gov for details
Severity CRITICAL (9.0-10.0), HIGH (7.0-8.9), MEDIUM (4.0-6.9), LOW (0.1-3.9)
Installed Ver What's in your image right now
Fixed Ver The version that patches the CVE. (none) = no fix available yet

Making Trivy Useful in CI

# Fail the build on critical vulnerabilities
trivy image --exit-code 1 --severity CRITICAL myapp:latest

# Ignore vulnerabilities with no fix (they're noise in CI)
trivy image --exit-code 1 --severity CRITICAL,HIGH --ignore-unfixed myapp:latest

# Also scan for secrets baked into image layers
trivy image --scanners vuln,secret myapp:latest

# Scan your Dockerfile for misconfigurations
trivy config Dockerfile

# Output JSON for automation
trivy image --format json --output results.json myapp:latest

Gotcha: Scanners only catch known vulnerabilities — things that already have a CVE. They cannot detect intentionally malicious code in packages that haven't been flagged yet. The event-stream attack, the xz backdoor, and typosquatting packages all fly under the scanner radar. Scanning is necessary but not sufficient.

Name Origin: Trivy was created by Teppei Fukuda as a weekend side project in 2019. The name comes from "trivial" — making security scanning trivially easy to set up. Aqua Security hired Fukuda and made Trivy their official open-source scanner. By 2023, it was the default scanner in Amazon ECR, Harbor, GitLab, and dozens of CI platforms.

Grype: The Alternative

# Scan an image
grype myapp:latest

# Fail on high severity
grype myapp:latest --fail-on high

# Scan from an SBOM (no Docker daemon needed)
syft packages myapp:latest -o spdx-json > sbom.json
grype sbom:sbom.json

Trivia: Running Trivy, Grype, and Snyk on the same image often produces different results. They consult different CVE databases, handle vendor-specific advisories differently (Debian, Alpine, and Red Hat each assess CVEs independently), and use different matching logic. For high-security environments, running multiple scanners is recommended.

Flashcard Check #2

Question Answer
What does trivy image --exit-code 1 --severity CRITICAL do? Scans the image for vulnerabilities and exits with code 1 (failing the CI build) if any CRITICAL-severity CVEs are found.
Why is --ignore-unfixed useful in CI but dangerous as a permanent policy? It reduces noise by hiding CVEs with no available fix, but a CVE might gain a fix later while your suppression persists. Run periodic full scans without this flag.
Can image scanning detect the xz backdoor or event-stream attack? No. Scanners detect known CVEs, not intentionally malicious code that hasn't been flagged. These attacks require different defenses: lockfile pinning, dependency review, SLSA provenance.

Part 6: SBOM — The Ingredient List

When Log4Shell hit in December 2021, the first question every organization asked was: "Are we affected?" Organizations without an SBOM spent days or weeks auditing services. Those with SBOMs queried a database and had answers in minutes.

An SBOM (Software Bill of Materials) is a machine-readable inventory of every component in your artifact: libraries, versions, licenses, and where they came from.

# Generate an SBOM with syft (Anchore's tool)
syft packages myapp:latest -o spdx-json > sbom.spdx.json

# What's inside?
cat sbom.spdx.json | jq '.packages[] | {name: .name, version: .versionInfo}' | head -20
# {"name": "adduser", "version": "3.134"}
# {"name": "apt", "version": "2.6.1"}
# {"name": "base-files", "version": "12.4+deb12u5"}
# {"name": "bash", "version": "5.2.15-2+b2"}
# ...

# Now scan that SBOM for vulnerabilities
grype sbom:sbom.spdx.json

# Generate SBOM in CycloneDX format (alternative standard)
syft packages myapp:latest -o cyclonedx-json > sbom.cdx.json

The Two SBOM Formats

SPDX CycloneDX
Created by Linux Foundation (2010) OWASP (2017)
ISO standard ISO/IEC 5962:2021 Yes (Ecma International)
Focus License compliance + security Security + vulnerability tracking
Tooling syft, Trivy, spdx-tools syft, Trivy, cdxgen
Use when Federal compliance, license auditing Vulnerability management, security-first

Both work. Pick one and be consistent.

Name Origin: SPDX stands for Software Package Data Exchange. CycloneDX was named by OWASP's Steve Springett — "Cyclone" for the speed of generating BOM data, "DX" for Developer Experience. SBOM itself borrows the term "Bill of Materials" from manufacturing, where a BOM lists every component in a physical product — every bolt, wire, and chip.

Trivia: SBOMs were mandated for US federal software by President Biden's May 2021 Executive Order on Improving the Nation's Cybersecurity. If you sell software to the US government, you must provide an SBOM. This drove enterprise adoption faster than any technical argument could.

Part 7: Signing and Verification — Proving What You Built

Scanning finds known vulnerabilities. SBOMs inventory components. But neither proves that the image you're about to deploy is the same one your CI pipeline built. That's what signing does.

The Cosign Workflow

Cosign (part of Sigstore) signs container images so consumers can verify their origin.

# Step 1: Build and push, capturing the immutable digest
DIGEST=$(docker build --push -t ghcr.io/myorg/myapp:v1.0.0 . \
  2>&1 | grep "digest:" | awk '{print $NF}')

# Step 2: Sign by digest using keyless OIDC (recommended for CI)
cosign sign --yes ghcr.io/myorg/myapp:v1.0.0@${DIGEST}

# Step 3: Consumers verify before deploying
cosign verify \
  --certificate-identity-regexp "https://github.com/myorg/myapp/.*" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/myorg/myapp:v1.0.0@${DIGEST}

What just happened: 1. Build produced an image with a specific digest (content hash) 2. Cosign requested a short-lived certificate from Fulcio (Sigstore's CA), tied to your CI identity (e.g., GitHub Actions OIDC token) 3. The signature was recorded in Rekor (Sigstore's transparency log) — a tamper-evident, append-only public ledger 4. Verification checks that the signature matches the digest AND that the signing identity matches your expected CI workflow

Under the Hood: Keyless signing means no long-lived private key to manage, rotate, or accidentally commit to git. Instead, cosign uses OIDC (OpenID Connect) — the same protocol behind "Sign in with Google." Your CI system proves its identity to Fulcio, which issues a short-lived certificate (valid for ~10 minutes). The signature is timestamped and logged in Rekor, so even after the certificate expires, the signature remains verifiable.

Name Origin: Sigstore combines "signature" and "store." Rekor is named after a town in Iceland, following a naming convention from the project's early development. Fulcio (the certificate authority) is named after Fulcio of Ravenna, a medieval Italian bishop. The project was started by Dan Lorenc and others at Google, donated to the OpenSSF (Open Source Security Foundation) in 2021.

Gotcha: Always sign by digest, not by tag. Tags are mutable — an attacker with registry write access can push a different image to the same tag. The signature would then point to the old (legitimate) image digest, while the tag points to the new (malicious) image. Consumers who verify by tag get a false sense of security. Deploy by digest.

Part 8: Admission Controllers — The Cluster Gatekeeper

Scanning, SBOMs, and signing are useless if nothing enforces them. Admission controllers are the enforcement layer in Kubernetes — they intercept every pod creation request and can reject it if it doesn't meet your policies.

Kyverno: Image Verification Policy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce    # Reject violations (not just log)
  rules:
    - name: verify-cosign-signature
      match:
        any:
          - resources:
              kinds: [Pod]
      verifyImages:
        - imageReferences:
            - "ghcr.io/myorg/*"
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/myorg/myapp/.github/workflows/release.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"

This says: every pod using an image from ghcr.io/myorg/* must have a valid cosign signature from our specific GitHub Actions release workflow. Everything else is rejected.

# Test it — try deploying an unsigned image
kubectl run test --image=ghcr.io/myorg/myapp:unsigned
# Error: admission webhook "validate.kyverno.svc" denied the request

# Deploy the signed image — works
kubectl run test --image=ghcr.io/myorg/myapp:v1.0.0@sha256:abc123...
# pod/test created

OPA Gatekeeper: The Alternative

OPA Gatekeeper uses Rego policies instead of Kyverno's YAML-based approach. More flexible, steeper learning curve:

# Deny images without a cosign signature annotation
package k8s.admission

deny[msg] {
    input.request.kind.kind == "Pod"
    container := input.request.object.spec.containers[_]
    not has_signature(container.image)
    msg := sprintf("Image %v must be signed", [container.image])
}
Kyverno OPA Gatekeeper
Policy language YAML (Kubernetes-native) Rego (general-purpose policy language)
Learning curve Low — if you know Kubernetes YAML Higher — Rego is its own language
Image verification Built-in verifyImages Requires external webhook
Best for Teams that want Kubernetes-native policies Organizations with complex, cross-system policy needs

Gotcha: Installing Kyverno or Gatekeeper in audit mode (validationFailureAction: Audit) and forgetting to switch to Enforce is one of the most common supply chain security failures. Set a deadline — 1-2 weeks — to resolve violations and flip the switch. Audit mode that never becomes enforcement is security theater.

Part 9: Pinning Everything — The Dependency Lockdown

The theme of this entire lesson is: mutable references are the enemy. Tags, version ranges, unpinned actions — anything that can silently change between builds is a vector.

Container Base Images

# BAD: mutable tag
FROM python:3.11-slim

# GOOD: pinned digest
FROM python:3.11-slim@sha256:2bac43769ace90ebd3ad83e5392295e25dfc58e58543d3ab326c3330b505283c

Use Renovate or Dependabot to automatically create PRs when new digests are available. Pinning doesn't mean never updating — it means updating intentionally.

npm

# BAD: installs whatever matches the semver range
npm install

# GOOD: installs exactly what's in the lockfile
npm ci

npm ci deletes node_modules/ and installs from package-lock.json exactly. It fails if the lockfile is out of sync with package.json. This is what your CI should use.

Python

# Generate a hash-pinned requirements file
pip-compile --generate-hashes requirements.in > requirements.txt

# Install with hash verification
pip install --require-hashes -r requirements.txt

With --require-hashes, pip verifies the SHA-256 hash of every downloaded package against the hash in requirements.txt. A republished package with different content (same version string, different bytes) will fail to install.

Go

Go's go.sum file already contains cryptographic hashes of every dependency. go mod verify checks them:

go mod verify
# all modules verified

GitHub Actions

# BAD: mutable tag
- uses: actions/checkout@v4

# GOOD: immutable SHA with version comment
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1

Dependency Confusion Prevention

# .npmrc — force all @myorg packages to come from your private registry
@myorg:registry=https://npm.myorg.internal/

# pip.conf — use ONLY the private index, no fallback to PyPI
[global]
index-url = https://pypi.myorg.internal/simple/

Use --index-url (not --extra-index-url) for pip. The --extra-index-url flag adds a fallback to the public registry, which is exactly what dependency confusion exploits.

Remember: The mnemonic for supply chain defense layers is SAVD: SBOM (know what you shipped), Attestation (prove how it was built), Verification (check before deploy), Dependency scanning (catch known vulns). Each layer catches different classes of attack. You need all four.

Part 10: The SLSA Framework — Maturity Model for Supply Chain Security

SLSA (Supply-chain Levels for Software Artifacts, pronounced "salsa") gives you a graduated path from "no security" to "hardened build platform."

Level Requirements What It Proves
Build L0 Nothing documented Nothing
Build L1 Provenance exists (build metadata) Someone documented how it was built
Build L2 Hosted build, signed provenance A known build service produced it
Build L3 Hardened build platform, non-falsifiable provenance The build environment itself is tamper-resistant

Most organizations are between L0 and L1. Getting to L2 (hosted builds with signed provenance) is the highest-value jump.

# GitHub Actions: generate SLSA provenance with the official generator
provenance:
  needs: build
  uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v1.9.0
  with:
    image: ghcr.io/${{ github.repository }}
    digest: ${{ needs.build.outputs.digest }}
  permissions:
    actions: read
    id-token: write
    packages: write

Name Origin: SLSA originated from Google's internal Binary Authorization for Borg (BAB), which gates what binaries can run on Google's infrastructure. It was open-sourced through the OpenSSF in 2021. The "salsa" pronunciation was chosen deliberately to make the framework more approachable and memorable.

Gotcha: SLSA Level 1 only requires documentation of provenance — it does not require signature verification. Many teams claim "SLSA compliance" at L1 thinking it provides strong security guarantees. The real defensive value starts at L2, where provenance is signed by a hosted build service and can be independently verified.

Part 11: Putting It All Together — The Full Pipeline

Here's what a complete supply chain security pipeline looks like in GitHub Actions:

name: Secure Build Pipeline
on:
  push:
    tags: ['v*']

jobs:
  build-scan-sign:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write          # Required for keyless signing
    steps:
      # Pin actions by SHA
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1

      - name: Build and push image
        id: build
        run: |
          DIGEST=$(docker build --push -t ghcr.io/${{ github.repository }}:${{ github.ref_name }} . \
            2>&1 | grep "digest:" | awk '{print $NF}')
          echo "digest=${DIGEST}" >> "$GITHUB_OUTPUT"

      - name: Generate SBOM
        run: syft ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }} -o spdx-json > sbom.spdx.json

      - name: Scan for vulnerabilities
        run: grype sbom:sbom.spdx.json --fail-on high

      - name: Sign image (keyless)
        run: cosign sign --yes ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}

      - name: Attest SBOM
        run: cosign attest --type spdxjson --predicate sbom.spdx.json --yes ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}

The flow: Build -> Generate SBOM -> Scan SBOM for vulns -> Sign image -> Attach SBOM attestation. If scanning fails, the image is never signed, and the admission controller will reject it.

Exercises

Exercise 1: Read a Trivy Report (2 minutes)

Given this Trivy output:

myapp:latest (debian 12.4)
Total: 5 (HIGH: 3, CRITICAL: 2)

│ openssl  │ CVE-2024-0727  │ CRITICAL │ 3.0.11-1  │ 3.0.13-1   │
│ curl     │ CVE-2024-2398  │ CRITICAL │ 7.88.1    │ 7.88.1-10  │
│ zlib1g   │ CVE-2023-45853 │ HIGH     │ 1:1.2.13  │ (none)     │
│ libc6    │ CVE-2023-6246  │ HIGH     │ 2.36-9    │ 2.36-9+d12 │
│ ncurses  │ CVE-2023-50495 │ HIGH     │ 6.4-4     │ 6.4-4+d12  │

Questions: 1. Which vulnerabilities would you fix first? 2. What does (none) in the Fixed Ver column mean for zlib1g? 3. What single action would fix the most vulnerabilities?

Answers 1. The two CRITICALs: openssl (CVE-2024-0727) and curl (CVE-2024-2398). Both have fixes available. 2. No fix is published yet. You can't patch it by updating the package. Mitigate by assessing whether zlib1g is actually used by your application, or suppress with a documented `.trivyignore` entry. 3. Rebuild the image with an updated base image (`docker build --no-cache`). Most of these CVEs are in base image packages, and the base image maintainers have already released patches.

Exercise 2: Pin a GitHub Action (5 minutes)

Your workflow contains:

- uses: docker/build-push-action@v5
- uses: aquasecurity/trivy-action@master

Pin both to their current commit SHAs.

Hint Go to each repository on GitHub. Click on the tag/branch. Find the commit SHA for the release you want. Format: `uses: org/repo@ # ` You can also use: `gh api repos/docker/build-push-action/git/ref/tags/v5 --jq '.object.sha'`
Solution 
# Get the SHA for docker/build-push-action v5
gh api repos/docker/build-push-action/git/ref/tags/v5 --jq '.object.sha'

# Get the SHA for trivy-action master (use latest release tag instead)
gh api repos/aquasecurity/trivy-action/git/ref/tags/master --jq '.object.sha'
Then update your workflow: 
- uses: docker/build-push-action@<sha-from-above>  # v5
- uses: aquasecurity/trivy-action@<sha-from-above>  # v0.x.x
Better: never pin to `master` — find the latest release tag and pin that SHA.

Exercise 3: Trace the Blast Radius (15 minutes)

Your organization runs 40 microservices. A CVE is published for libexpat version 2.5.0. You need to determine which services are affected.

Design the process: what tools would you use, what commands would you run, and what would the workflow look like?

Solution approach 1. **If you have SBOMs stored per image:** 
# Search all SBOMs for the affected package
for sbom in sboms/*.spdx.json; do
  if jq -e '.packages[] | select(.name == "libexpat" and .versionInfo == "2.5.0")' "$sbom" > /dev/null 2>&1; then
    echo "AFFECTED: $sbom"
  fi
done
2. **If you don't have SBOMs (the panic path):** 
# Scan every running image in the cluster
kubectl get pods -A -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | \
  sort -u | while read img; do
    trivy image --severity CRITICAL,HIGH "$img" 2>/dev/null | grep -q "libexpat" && echo "AFFECTED: $img"
  done
3. **Remediation:** Rebuild affected images with an updated base image that patches libexpat. Sign the new images. The admission controller ensures only signed images are admitted. The SBOM path takes seconds. The no-SBOM path takes minutes to hours depending on cluster size and registry performance. This is why you generate SBOMs.

Cheat Sheet

Tool What It Does Key Command
Trivy Scans images for CVEs trivy image --exit-code 1 --severity CRITICAL myapp:latest
Grype Alternative image/SBOM scanner grype sbom:sbom.spdx.json --fail-on high
syft Generates SBOMs syft packages myapp:latest -o spdx-json > sbom.json
cosign Signs and verifies images cosign sign --yes image@digest / cosign verify ...
Rekor Transparency log for signatures rekor-cli search --sha <digest>
Kyverno K8s admission controller validationFailureAction: Enforce + verifyImages
Gatekeeper K8s admission controller (Rego) OPA-based policy enforcement
Renovate Automates dependency updates Pins digests, creates update PRs
npm ci Lockfile-exact install Fails if lockfile is out of sync
pip --require-hashes Hash-verified pip install Rejects packages with wrong hash
go mod verify Verifies go.sum hashes Fails if any module was tampered with
Defense Catches Doesn't Catch
Vulnerability scanning Known CVEs Zero-days, malicious code without CVEs
SBOM "Do we have X?" queries Whether X is actually malicious
Image signing Tampering after build Compromised build environment
Admission controllers Unsigned/unverified images Policy misconfiguration
Lockfile pinning Dependency version drift Malicious new versions
Hash pinning Republished-same-version attacks Malicious new versions
SLSA L3 provenance Build pipeline tampering Source code compromise

Takeaways

  • Your application is mostly other people's code. The median app has 500+ open-source dependencies. Each one is a trust decision you probably didn't make consciously.

  • Tags lie, digests don't. Mutable tags (:latest, :v1.2.3, @main) can point to different content tomorrow. Digests are content-addressed and immutable. Pin by digest everywhere: base images, deployed images, GitHub Actions.

  • Scanning is necessary but not sufficient. Scanners catch known CVEs. They cannot catch intentionally malicious code (typosquatting, account takeover, social engineering). Defense requires layers: scanning + SBOMs + signing + admission control + pinning.

  • SBOMs turn days of panic into minutes of queries. When the next Log4Shell hits, you want to answer "are we affected?" by searching a database, not by auditing 40 services by hand.

  • Audit mode is not a destination. Admission controllers in audit mode that never switch to enforce are security theater. Set a deadline and flip the switch.

  • The attacker's time horizon is longer than yours. The xz backdoor took two years of social engineering. Supply chain security isn't a project — it's a permanent practice.