Investigation: Container Image Vuln Scanner False Positive, Blocks Deploy Pipeline¶
Phase 1: Security Investigation (Dead End)¶
Check the CVE details:
$ trivy image --severity CRITICAL registry.internal:5000/notification-service:v3.8.1
registry.internal:5000/notification-service:v3.8.1 (debian 12.4)
Total: 1 (CRITICAL: 1)
┌───────────────┬────────────────┬──────────┬──────────────┬──────────────────┐
│ Library │ Vulnerability │ Severity │ Installed │ Fixed Version │
├───────────────┼────────────────┼──────────┼──────────────┼──────────────────┤
│ libcurl4 │ CVE-2026-1234 │ CRITICAL │ 8.5.0-2 │ 8.5.0-2+deb12u2 │
└───────────────┴────────────────┴──────────┴──────────────┴──────────────────┘
Check the NVD entry:
$ curl -s "https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2026-1234" | jq '.vulnerabilities[0].cve.descriptions[0].value'
"A remote code execution vulnerability in libcurl when processing specially crafted HTTP/2 PUSH responses. Affects libcurl 8.4.0 through 8.5.0 when built with HTTP/2 support (nghttp2)."
The CVE affects libcurl 8.4.0-8.5.0 with HTTP/2 support. Check if the application uses HTTP/2:
# Check the notification-service code
$ grep -r "http2\|HTTP/2\|h2" notification-service/src/
# (no matches — the application does not use HTTP/2)
# Check if libcurl in the image was built with HTTP/2
$ docker run --rm registry.internal:5000/notification-service:v3.8.1 \
curl --version | head -3
curl 8.5.0 (x86_64-pc-linux-gnu) libcurl/8.5.0 OpenSSL/3.0.13 zlib/1.2.13 nghttp2/1.59.0
Release-Date: 2024-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps ...
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy ...
libcurl was built with nghttp2 (HTTP/2 support). Technically, the library is vulnerable. But the application never makes HTTP/2 requests — it only uses HTTP/1.1 for webhook deliveries. The vulnerability requires processing HTTP/2 PUSH responses from a malicious server, and the notification service only sends outbound webhooks to pre-configured URLs.
The Pivot¶
Despite the low exploitability, Trivy correctly identifies the vulnerable library version. But check the Debian security tracker:
$ curl -s "https://security-tracker.debian.org/tracker/CVE-2026-1234"
# Status: fixed in 8.5.0-2+deb12u2
# Check if our base image has the fix
$ docker run --rm registry.internal:5000/notification-service:v3.8.1 \
dpkg -l libcurl4
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-============-==============-============-===============================
ii libcurl4 8.5.0-2+deb12u2 amd64 easy-to-use client-side URL library
Wait — the installed version is 8.5.0-2+deb12u2, which is the fixed version. Trivy is reporting the CVE for 8.5.0-2 but the image actually has 8.5.0-2+deb12u2 installed.
Phase 2: DevOps Tooling Investigation (Root Cause)¶
Check why Trivy is seeing the wrong version:
# Check Trivy's database age
$ trivy --version
Version: 0.49.1
Vulnerability DB:
Version: 2
UpdatedAt: 2026-03-19T06:00:00Z
NextUpdate: 2026-03-20T06:00:00Z
DownloadedAt: 2026-03-19T08:15:00Z
The Trivy database is current (updated this morning). But check the image layers:
# Check when the base image was rebuilt
$ docker inspect registry.internal:5000/notification-service:v3.8.1 | jq '.[0].Created'
"2026-03-19T14:10:00Z"
# Check the Dockerfile
$ cat notification-service/Dockerfile
FROM python:3.11-slim-bookworm
# ... application setup
The base image python:3.11-slim-bookworm was pulled during the build. But the CI pipeline caches the base image:
# .github/workflows/ci.yml
- name: Pull cached base image
run: docker pull registry.internal:5000/base-images/python:3.11-slim-bookworm || true
The CI pipeline is pulling a cached copy of the base image from the internal registry, not the latest from Docker Hub. The cached copy has libcurl4 8.5.0-2 (without the security patch), but when the image is built, apt-get upgrade during the build step updates it to 8.5.0-2+deb12u2. Trivy scans the image manifest, which includes layer metadata from the base image that still references the old version. The actual installed package is patched, but Trivy reads the OS package database from a cached layer.
# Verify: Trivy is reading the wrong dpkg status
$ trivy image --list-all-pkgs registry.internal:5000/notification-service:v3.8.1 | grep libcurl
libcurl4 8.5.0-2 8.5.0-2+deb12u2 CVE-2026-1234
# The "Installed" column shows 8.5.0-2 (from the base layer's dpkg status)
# but the actual running version is 8.5.0-2+deb12u2 (updated in a later layer)
The internal base image cache has not been refreshed. Trivy's SBOM extraction reads the package database from the earliest layer, not the final squashed state, due to a known Trivy issue with multi-layer dpkg status files.
Domain Bridge: Why This Crossed Domains¶
Key insight: The symptom was a security scanner blocking deployments (security), the root cause was the CI pipeline's stale base image cache causing Trivy to misread the installed package version (devops_tooling), and the fix requires updating the Kubernetes-hosted build infrastructure to refresh the base image cache (kubernetes_ops). This is common because: container image scanning depends on accurate SBOM extraction from image layers. Build caches, multi-stage builds, and layer ordering can cause scanners to report stale package versions. The tension between security gates and deployment velocity is a chronic cross-domain friction point.
Root Cause¶
The CI pipeline caches a base image in the internal registry. This cached base image has an older version of libcurl (8.5.0-2). During the build, apt-get upgrade updates to the fixed version (8.5.0-2+deb12u2), but Trivy reads the package database from the base image layer (which still lists the old version) rather than the final merged state. This is a known Trivy issue with multi-layer Debian package databases, and the stale base image cache exacerbates it.