Container Registries: Where Your Images Actually Live

• lesson
• oci-distribution-spec
• registry-http-api
• content-addressable-storage
• tags-vs-digests
• registry-authentication
• multi-arch-manifests
• garbage-collection
• mirroring
• rate-limiting
• vulnerability-scanning
• image-promotion ---# Container Registries — Where Your Images Actually Live

Topics: OCI distribution spec, registry HTTP API, content-addressable storage, tags vs digests, registry authentication, multi-arch manifests, garbage collection, mirroring, rate limiting, vulnerability scanning, image promotion Level: L1–L2 (Foundations to Operations) Time: 60–90 minutes Prerequisites: None (everything is explained from scratch)

---

The Mission

It's 2:17 AM. PagerDuty fires. The payments service is returning 500s. You check the deployment — it rolled out 40 minutes ago, but nobody triggered a deploy. The Kubernetes Deployment spec hasn't changed. The Git repo hasn't changed. Yet the pods are running different code than they were two hours ago.

You dig in. The deployment references payments-api:latest. Someone on the platform team pushed a new image to that tag at 1:35 AM as part of a "routine base image update." When a node recycled at 1:42 AM, the kubelet pulled the new latest — and got the broken build. The other nodes still have the old image cached. Half your fleet is running the old code, half is running the new code. Your customers are getting different behavior depending on which pod handles their request.

This is not a Docker problem or a Kubernetes problem. This is a registry problem — and you cannot fix what you do not understand. This lesson traces the path an image takes from docker push to docker pull, through every HTTP request, manifest, and SHA256 digest. By the end, you'll know exactly why tags lie, how registries actually store your images, and how to build a promotion pipeline that prevents 2 AM surprises.

---

What Is a Container Registry, Really?

A container registry is an HTTP API that stores and serves OCI (Open Container Initiative) images. That's it. It is not magic. It is not a database. It is a content-addressable blob store with a metadata index on top.

You (docker push) → HTTPS → Registry API → Blob Storage (layers, configs, manifests)
You (docker pull) → HTTPS → Registry API → Blob Storage → Your Local Daemon

Every registry — Docker Hub, Amazon ECR, Google Artifact Registry, Harbor, GitHub Container Registry — implements the same OCI Distribution Specification. The spec defines the HTTP endpoints, the manifest format, and how content addressing works.

Name Origin: The OCI (Open Container Initiative) was founded in June 2015 by Docker and CoreOS under the Linux Foundation. Docker donated its image format and runtime specs as the starting point. The word "initiative" was chosen carefully — it was a diplomatic ceasefire in the Docker-vs-rkt container wars. The OCI Distribution Spec (the registry protocol) was formalized later, based on the Docker Registry V2 API.

Trivia: Docker Hub serves over 13 billion image pulls per month as of 2023, making it one of the highest-traffic package registries in the world. When Docker imposed rate limits in November 2020, it broke CI pipelines globally and accelerated adoption of GitHub Container Registry and ECR Public.

---

What Happens When You docker pull

Let's trace a real pull, HTTP request by HTTP request. You type:

docker pull registry.example.com/payments/api:v2.4.1

Here is every step the Docker daemon takes.

Step 1: Resolve the tag to a manifest

GET /v2/payments/api/manifests/v2.4.1
Host: registry.example.com
Accept: application/vnd.oci.image.manifest.v1+json,
        application/vnd.docker.distribution.manifest.v2+json,
        application/vnd.oci.image.index.v1+json

The registry returns a manifest — a JSON document that describes the image:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:a1b2c3d4...",
    "size": 7023
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:e4f5a6b7...",
      "size": 32654947
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:c8d9e0f1...",
      "size": 16724
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:2a3b4c5d...",
      "size": 73109
    }
  ]
}

That manifest is the image's bill of materials. It says: "this image is made of a config blob at this digest, and three layers at these digests." Every digest is a SHA256 hash of the content it points to.

Step 2: Download the config blob

GET /v2/payments/api/blobs/sha256:a1b2c3d4...
Host: registry.example.com

The config contains metadata: environment variables, entrypoint, working directory, architecture, OS, and the ordered list of layer diff IDs. This is where Docker learns how to run the image.

Step 3: Download each layer (in parallel)

GET /v2/payments/api/blobs/sha256:e4f5a6b7...
GET /v2/payments/api/blobs/sha256:c8d9e0f1...
GET /v2/payments/api/blobs/sha256:2a3b4c5d...

Layers are downloaded in parallel. If a layer already exists locally (because another image shares the same base layer), it is skipped entirely. This is why pulling a new version of python:3.12-slim is fast — most of the base layers are already on disk.

Step 4: Verify and unpack

For each downloaded blob, Docker verifies the SHA256 hash against the digest in the manifest. If even one byte is different, the pull fails. Then the layers are unpacked into the local storage driver (typically overlay2) and stacked in order.

Layer 1 (base OS)         ← sha256:e4f5a6b7...
Layer 2 (pip install)     ← sha256:c8d9e0f1...
Layer 3 (COPY app code)   ← sha256:2a3b4c5d...

Under the Hood: Content-addressable storage means the blob's name IS its content hash. You cannot tamper with a blob without changing its digest, which would break the manifest reference. This is the same principle behind Git objects — both use SHA256 to make content immutable and verifiable. If the registry serves a corrupted layer, the client rejects it.

See it yourself

# Inspect a manifest without pulling the full image
docker manifest inspect python:3.12-slim

# Using crane (lightweight registry tool from Google)
crane manifest python:3.12-slim | jq '.layers[] | {digest, size}'

# Using skopeo (Red Hat's registry tool — no daemon required)
skopeo inspect docker://docker.io/library/python:3.12-slim

Tool	Needs Docker daemon?	Can copy between registries?	Best for
docker manifest	Yes	No	Quick inspection
crane	No	Yes	CI pipelines, scripting
skopeo	No	Yes	Air-gapped copies, multi-registry
curl + jq	No	No	Learning the raw API

---

Flashcard Check #1

Cover the answers with your hand. Test yourself.

Question	Answer
What are the three components an OCI manifest references?	Config blob, ordered layer blobs, and their SHA256 digests
Why can layers be skipped during docker pull?	If the layer's SHA256 digest already exists in local storage, it's identical content — no need to re-download
What HTTP endpoint resolves a tag to a manifest?	GET /v2/<name>/manifests/<reference>
What happens if a downloaded blob's hash doesn't match the manifest?	The pull fails — content-addressable storage guarantees integrity

---

Why Tags Lie and Digests Don't

This is the single most important concept in this lesson. Get this wrong and you will have a 2 AM incident like the one in the mission.

Tags are mutable pointers

A tag is a human-readable name like v2.4.1 or latest that points to a manifest digest. The registry stores this as a simple mapping:

v2.4.1  →  sha256:abcdef123456...
latest  →  sha256:abcdef123456...

Anyone with push access can overwrite that mapping at any time:

# Monday: push v1 code as "latest"
docker push myregistry.com/app:latest    # now points to sha256:aaa...

# Tuesday: push v2 code as "latest"
docker push myregistry.com/app:latest    # now points to sha256:bbb...

After Tuesday's push, latest points to completely different bytes. Any system that pulls app:latest will get v2 — whether it expected it or not.

Digests are immutable identifiers

A digest is the SHA256 hash of the manifest itself. It cannot be changed without changing the content. To reference an image immutably:

# This will always pull the exact same bytes, forever
docker pull myregistry.com/app@sha256:a1b2c3d4e5f6...

No one can overwrite a digest. If the content changes, the digest changes. Period.

The practical difference

# Mutable: who knows what you'll get?
image: payments-api:latest

# Slightly better but still mutable:
image: payments-api:v2.4.1

# Immutable: guaranteed to be the exact bytes you tested
image: payments-api@sha256:a1b2c3d4e5f678901234567890abcdef...

Gotcha: Even "version" tags like v2.4.1 are mutable. There is nothing in the registry protocol that prevents someone from pushing different content to the same tag. The only truly immutable reference is a digest. Treat version tags as conventions enforced by process, not by technology.

War Story: In the mission scenario, the latest tag was overwritten by an automated base image update. But this pattern has caused larger incidents. In one widely reported case, a team's CI pipeline pushed to a :release tag that production deployments referenced. A developer accidentally triggered the pipeline from a feature branch. The tag now pointed to untested code. Half the fleet pulled the new image on pod restarts over the next hour. The rollback required manually re-tagging the old digest and forcing a rolling restart across all nodes. Total time to resolution: 3 hours. Root cause: mutable tags in production.

How to get the digest of an image

# From a local image
docker inspect --format='{{index .RepoDigests 0}}' payments-api:v2.4.1

# From a remote registry (without pulling)
crane digest myregistry.com/payments-api:v2.4.1

# From skopeo
skopeo inspect docker://myregistry.com/payments-api:v2.4.1 | jq -r '.Digest'

# From the Docker CLI
docker manifest inspect myregistry.com/payments-api:v2.4.1 | jq -r '.digest // .config.digest'

Mental Model: Think of tags like DNS names and digests like IP addresses. google.com might point to different servers at different times — the name is a mutable convenience. But 142.250.80.46 is always the same host. When reliability matters, you use the address. When convenience matters, you use the name. In production, you need the address.

---

Registry Authentication: Who Gets to Pull?

Every registry has its own authentication dance, but the protocol follows a common pattern defined in the Docker Registry Token Authentication spec.

The authentication flow

The client sends a request, gets a 401 with a WWW-Authenticate: Bearer header pointing to a token endpoint, fetches a short-lived token with credentials, then retries with Authorization: Bearer <token>. This handshake is why the first pull to a new registry feels slow.

Registry-specific authentication

Registry	Auth method	Credential source
Docker Hub	Username/password or token	docker login, ~/.docker/config.json
Amazon ECR	AWS IAM + short-lived token	aws ecr get-login-password (valid 12h)
Google GAR	OAuth2 or service account	gcloud auth configure-docker
GitHub GHCR	Personal access token	echo $GITHUB_TOKEN \| docker login ghcr.io -u USERNAME --password-stdin
Harbor	LDAP/OIDC/local users	docker login harbor.company.com

# ECR login (AWS) — token expires in 12 hours
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com

# Google Artifact Registry
gcloud auth configure-docker us-docker.pkg.dev

# GitHub Container Registry
echo $GITHUB_TOKEN | docker login ghcr.io -u myuser --password-stdin

Gotcha: ECR tokens expire after 12 hours. If your CI pipeline caches the Docker credential config, builds start failing at 3 AM when the cached token expires. Use amazon-ecr-credential-helper for automatic refresh, or re-authenticate at the start of every CI job.

In Kubernetes, registry credentials are stored as imagePullSecrets — a Secret of type kubernetes.io/dockerconfigjson containing the base64-encoded ~/.docker/config.json, referenced in the Pod spec's imagePullSecrets field.

---

Multi-Architecture Images: One Tag, Many Platforms

Your team runs amd64 servers in AWS and arm64 Mac laptops for development. You want docker pull myapp:v2.4.1 to get the right binary on both. This is what manifest lists (also called image indexes) solve.

How it works

Instead of a manifest pointing directly to layers, the tag points to a manifest list that contains pointers to platform-specific manifests:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:aaa111...",
      "size": 7143,
      "platform": { "architecture": "amd64", "os": "linux" }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:bbb222...",
      "size": 7143,
      "platform": { "architecture": "arm64", "os": "linux" }
    }
  ]
}

When Docker resolves the tag, it checks the manifest list, finds the entry matching the local platform, then follows that digest to the actual manifest and layers.

# Inspect a multi-arch image
docker manifest inspect python:3.12-slim
# Shows entries for amd64, arm64, arm/v7, etc.

# Build and push multi-arch
docker buildx create --use
docker buildx build --platform linux/amd64,linux/arm64 \
  -t myregistry.com/myapp:v2.4.1 --push .

Under the Hood: Multi-arch images work because the tag → manifest indirection has an extra level. Single-arch: tag → manifest → layers. Multi-arch: tag → manifest list → (per-platform manifest) → layers. The Docker client picks the right manifest automatically. Tools like crane and skopeo can inspect all platforms without pulling.

---

Flashcard Check #2

Question	Answer
What is the relationship between a tag and a digest?	A tag is a mutable pointer to a digest. A digest is the SHA256 hash of the manifest content — immutable by definition.
Why is image: myapp:v2.4.1 still technically mutable?	Nothing in the registry protocol prevents overwriting a tag. Any push to the same tag replaces the pointer.
How does multi-arch resolution work?	Tag resolves to a manifest list (image index). The client selects the manifest matching its platform (arch + OS), then pulls those layers.
What happens when an ECR token expires mid-pipeline?	Pulls and pushes fail with 401 Unauthorized. Use the credential helper for auto-refresh.

---

The Docker Hub Rate Limiting Problem

In November 2020, Docker Hub began enforcing pull rate limits:

Account type	Rate limit
Anonymous (no login)	100 pulls / 6 hours per IP
Authenticated free	200 pulls / 6 hours per user
Docker Pro / Team	5,000 pulls / day
Docker Business	Unlimited

For a single developer, this is fine. For a CI system running 50 builds an hour on shared infrastructure where many projects pull from Docker Hub — anonymous limits are exhausted in under two hours.

Your builds start failing intermittently with toomanyrequests errors. Some succeed (rate limit window reset), some don't. The failures are non-deterministic.

Solutions

# 1. Authenticate to Docker Hub (doubles your limit)
echo $DOCKERHUB_TOKEN | docker login -u myuser --password-stdin

# 2. Mirror images into your own registry
skopeo copy \
  docker://docker.io/library/python:3.12-slim \
  docker://myregistry.com/mirrors/python:3.12-slim

# 3. Use a pull-through cache (Harbor, Nexus, or registry mirror)
# Configure Docker daemon to use a mirror:
# /etc/docker/daemon.json
{
  "registry-mirrors": ["https://mirror.mycompany.com"]
}

# 4. Use alternative registries that don't rate-limit
# GitHub Container Registry (ghcr.io)
# Amazon ECR Public (public.ecr.aws)
# Google's mirror (mirror.gcr.io)
# Quay.io

Trivia: Docker Hub's rate limiting announcement in November 2020 broke CI pipelines worldwide overnight. GitHub Container Registry (ghcr.io) launched its general availability just weeks later, and Amazon ECR Public followed shortly after. The rate limit event single-handedly accelerated the multi-registry ecosystem.

---

Registry Mirroring and Pull-Through Caches

A pull-through cache sits between your infrastructure and the upstream registry. The first pull fetches from upstream and caches locally. Subsequent pulls are served from the cache, which is faster and immune to upstream rate limits or outages.

Pod → kubelet → pull-through cache → (cache miss?) → upstream registry
                     ↓ (cache hit)
                  local blob store

Harbor, Nexus, and the built-in Docker registry all support pull-through caching. Configure once, then pull through the cache: docker pull harbor.company.com/dockerhub-cache/library/nginx:1.25. The cache stores layers locally. Every subsequent pull is local-speed.

Why mirroring matters for production

• Availability: If Docker Hub is down (it has happened), your deploys still work
• Speed: Local cache is 10-100x faster than cross-internet pulls
• Rate limits: Only the cache talks to upstream, not every node
• Air-gapped environments: The mirror is the only option

Mental Model: A pull-through cache is to container images what a CDN is to web content. You would never serve your website's static assets directly from a single origin server. The same logic applies to container images — cache close to the consumer.

---

Registry Garbage Collection

Registries accumulate dead weight. Every time you push a new image with the same tag, the old manifest becomes unreferenced — but the blobs remain. This is because registries use content-addressable storage: blobs are stored by digest, and multiple manifests can reference the same blob.

What garbage collection does

Before GC:
  manifest A (tagged) → blob 1, blob 2, blob 3
  manifest B (untagged) → blob 2, blob 4, blob 5

  Blobs stored: 1, 2, 3, 4, 5

After GC (mark and sweep):
  manifest A (tagged) → blob 1, blob 2, blob 3  ← kept
  manifest B (untagged) → deleted

  Blobs stored: 1, 2, 3  (blob 4, 5 swept because no manifest references them)
  Blob 2 is kept because manifest A still references it

Running GC on popular registries

# Docker Registry (self-hosted) — requires read-only mode
docker exec registry bin/registry garbage-collect /etc/docker/registry/config.yml

# Harbor — automatic GC scheduling in admin UI
# Or trigger via API:
curl -X POST "https://harbor.company.com/api/v2.0/system/gc/schedule" \
  -H "Content-Type: application/json" \
  -d '{"schedule": {"type": "Manual"}}'

# ECR — lifecycle policies handle this automatically
aws ecr put-lifecycle-policy --repository-name payments-api --lifecycle-policy-text '{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Keep last 10 images",
      "selection": {
        "tagStatus": "any",
        "countType": "imageCountMoreThan",
        "countNumber": 10
      },
      "action": { "type": "expire" }
    }
  ]
}'

Gotcha: The self-hosted Docker Registry (the registry:2 image) requires the registry to be in read-only mode during garbage collection, or you risk deleting blobs that a concurrent push is referencing. Harbor handles this automatically. ECR and GAR handle it behind the scenes. But if you run the open-source registry, you must coordinate GC with pushes.

---

Vulnerability Scanning at the Registry Level

Build-time scanning catches vulnerabilities when you build. But CVEs are discovered daily. An image that was clean last week might have three new CRITICALs today — and it is already running in production.

Registry-level scanning solves this by continuously rescanning stored images:

Registry	Scanning	Rescanning
Docker Hub	Docker Scout (included)	On push + continuous
ECR	Enhanced Scanning (Inspector)	On push + continuous
GAR / GCR	Artifact Analysis (on-demand)	On push + on-demand
Harbor	Trivy (built-in)	On push + scheduled
GHCR	Dependabot alerts	On push

# ECR: enable enhanced scanning
aws ecr put-registry-scanning-configuration \
  --scan-type ENHANCED \
  --rules '[{"repositoryFilters": [{"filter": "*", "filterType": "WILDCARD"}], "scanFrequency": "CONTINUOUS_SCAN"}]'

# Harbor: scan an image on demand
curl -X POST "https://harbor.company.com/api/v2.0/projects/myproject/repositories/myapp/artifacts/sha256:abc123/scan"

# Trivy: scan a remote image without pulling
trivy image --severity CRITICAL,HIGH myregistry.com/payments-api:v2.4.1

Under the Hood: Registry scanners extract the SBOM (Software Bill of Materials) from each image's layers — listing every OS package, language dependency, and binary. Then they match that SBOM against CVE databases (NVD, vendor advisories). When a new CVE is published, they re-match against all stored SBOMs without re-downloading layers. This is why continuous scanning is cheap once the initial SBOM is built.

---

Image Promotion Pipelines: Dev to Staging to Prod

In a mature setup, images don't just get pushed to one place. They flow through a pipeline:

Build → Dev Registry → (tests pass) → Staging Registry → (approval) → Prod Registry

Separate registries (or repositories) give you access control (only CI promotes to prod), blast radius containment, audit trails, and different scanning policies per environment.

Promotion by re-tagging (not rebuilding)

The key insight: you never rebuild between environments. You copy the exact same digest from one registry (or repository) to another. This guarantees that what you tested is what you deploy.

# Promote from dev to staging using crane
crane copy \
  dev-registry.company.com/payments-api@sha256:abc123... \
  staging-registry.company.com/payments-api:v2.4.1

# Promote using skopeo
skopeo copy \
  docker://dev-registry.company.com/payments-api@sha256:abc123... \
  docker://prod-registry.company.com/payments-api:v2.4.1

# Verify the digest is identical after promotion
crane digest dev-registry.company.com/payments-api@sha256:abc123...
crane digest prod-registry.company.com/payments-api:v2.4.1
# Both should output: sha256:abc123...

Remember: "Promote by digest, never by rebuild." If you rebuild for production, you get a different image — different timestamps, potentially different base layer versions, different dependency resolutions. The image you tested is not the image you deployed. Copy the bytes. Verify the digest.

---

Flashcard Check #3

Question	Answer
Why does registry garbage collection exist?	Untagged manifests and unreferenced blobs accumulate when tags are overwritten. GC reclaims storage by deleting orphaned content.
What is a pull-through cache?	A local proxy that fetches from upstream on first request, then serves from cache. Reduces latency, avoids rate limits, provides availability.
Why should you promote images by copying digests rather than rebuilding?	Rebuilding produces a different image (different timestamps, possibly different deps). Copying the digest guarantees byte-for-byte identity between tested and deployed.
What do Docker Hub anonymous rate limits look like in 2024?	100 pulls per 6 hours per IP address. Authenticated free accounts get 200 per 6 hours.

---

Cost Management: Storage and Egress

Registries are not free. The costs come from two sources:

Storage — every image version occupies space. A single Python app image at 150MB, pushed 20 times a day across 5 services, is 15GB of new storage per day. After a month without cleanup: 450GB.

Egress — pulling images costs money in cloud registries. ECR charges $0.09/GB for cross-region pulls. Pulling a 500MB image across 100 nodes in a different region: $4.50 per deployment. Do that 10 times a day: $45/day, $1,350/month.

How to control costs

Strategy	What it does	Impact
Lifecycle policies	Auto-delete images older than N days or keep only last N	Reduces storage 50-80%
Same-region pulls	Keep registry in the same region as your compute	Eliminates egress fees
Smaller images	Alpine/distroless instead of Ubuntu	5-10x less storage and egress
Layer sharing	Common base images shared across services	Reduces effective storage
Pull-through cache	Cache in each region	Eliminates cross-region egress

# ECR: check repository size
aws ecr describe-repositories --query 'repositories[*].[repositoryName]' --output text | \
  while read repo; do
    size=$(aws ecr describe-images --repository-name "$repo" \
      --query 'imageDetails[*].imageSizeInBytes' --output text | \
      awk '{s+=$1} END {printf "%.1f MB\n", s/1024/1024}')
    echo "$repo: $size"
  done

---

The War Room: A Tag Overwrite Postmortem

Let's close the loop on the mission. Here is what the investigation revealed and what the team changed.

Timeline

• 1:35 AM — Platform team's automated pipeline pushes a base image rebuild to payments-api:latest. A misconfigured variable defaulted the tag to latest.
• 1:42 AM — Node recycled by cluster autoscaler. Kubelet pulls payments-api:latest — gets the broken build. Other nodes still cache the old image.
• 1:43 AM — Payments service on the new node returns 500s (missing env var the new base image expects).
• 2:17 AM — PagerDuty fires. On-call starts investigation.
• 2:45 AM — kubectl get pods -o jsonpath='{range .items[*]}{.status.containerStatuses[0].imageID}{"\n"}{end}' shows two distinct digests across the fleet.
• 3:10 AM — Rollback: re-tag known-good digest to latest, force rolling restart.

What changed after

Before	After
image: payments-api:latest	image: payments-api@sha256:... (pinned by digest in Helm values)
No image promotion pipeline	Dev → Staging → Prod promotion using crane copy by digest
No tag immutability	ECR tag immutability enabled on production repositories
imagePullPolicy: Always	imagePullPolicy: IfNotPresent with digest pinning
No registry monitoring	Alerts on unexpected tag overwrites via registry webhooks

# Enable tag immutability on ECR (prevents overwrites)
aws ecr put-image-tag-mutability \
  --repository-name payments-api \
  --image-tag-mutability IMMUTABLE

Gotcha: Enabling tag immutability means you cannot push the same tag twice. This breaks workflows that rely on overwriting latest or stable. You need to adopt unique tags (Git SHA, build number, semver) before enabling immutability.

---

Exercises

Exercise 1: Inspect a real manifest (2 minutes)

# Inspect without pulling the full image
crane manifest nginx:1.25 | jq .
# Or: docker manifest inspect nginx:1.25

Count the layers and total their sizes. How many layers does nginx:1.25 have?

Exercise 2: Compare tag vs digest (5 minutes)

DIGEST=$(crane digest python:3.12-slim)
docker pull python:3.12-slim
docker pull python@$DIGEST
docker images --digests | grep python

Are the local image IDs identical? What would it mean if they were not?

Answer

They should be identical — the tag resolved to the same digest. If the tag was overwritten between the two pulls, they would differ. This demonstrates why digests are reliable.

Exercise 3: Build a promotion pipeline (15 minutes)

# Two local registries simulating dev and prod
docker run -d -p 5000:5000 --name dev-registry registry:2
docker run -d -p 5001:5000 --name prod-registry registry:2

# Build, push to dev, then promote by digest
docker build -t localhost:5000/myapp:v1 .
docker push localhost:5000/myapp:v1
DIGEST=$(crane digest localhost:5000/myapp:v1)
crane copy localhost:5000/myapp@$DIGEST localhost:5001/myapp:v1

# Verify digests match
crane digest localhost:5000/myapp:v1
crane digest localhost:5001/myapp:v1

The digests should be identical. You promoted the exact bytes, not a rebuild.

Exercise 4: Investigate rate limit headers (5 minutes)

# Check your Docker Hub rate limit status
TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/nginx:pull" | jq -r .token)
curl -s -I -H "Authorization: Bearer $TOKEN" \
  "https://registry-1.docker.io/v2/library/nginx/manifests/latest" | \
  grep -i ratelimit

Look for RateLimit-Limit and RateLimit-Remaining headers. How many pulls do you have left?

---

Cheat Sheet

Task	Command
Inspect manifest (no pull)	crane manifest <image> or docker manifest inspect <image>
Get image digest	crane digest <image>
Copy between registries	crane copy <src> <dst> or skopeo copy docker://<src> docker://<dst>
Inspect remote image metadata	skopeo inspect docker://<image>
List tags in a repo	crane ls <repo> or curl -s https://<registry>/v2/<repo>/tags/list
Check Docker Hub rate limit	curl with Bearer token, read RateLimit-Remaining header
ECR login	aws ecr get-login-password \| docker login --username AWS --password-stdin <account>.dkr.ecr.<region>.amazonaws.com
ECR enable tag immutability	aws ecr put-image-tag-mutability --image-tag-mutability IMMUTABLE
ECR lifecycle policy	aws ecr put-lifecycle-policy --lifecycle-policy-text file://policy.json
Scan remote image	trivy image <image>
Generate SBOM	trivy image --format cyclonedx -o sbom.json <image>

Key registry endpoints (OCI Distribution Spec)

Endpoint	Purpose
GET /v2/	API version check (also used as health check)
GET /v2/<name>/manifests/<ref>	Get manifest by tag or digest
PUT /v2/<name>/manifests/<ref>	Push a manifest
GET /v2/<name>/blobs/<digest>	Download a layer or config blob
POST /v2/<name>/blobs/uploads/	Start a blob upload
GET /v2/<name>/tags/list	List tags for a repository
DELETE /v2/<name>/manifests/<ref>	Delete a manifest (by digest only)

---

Takeaways

1. Tags are mutable pointers. Digests are immutable hashes. In production, pin by digest or enforce tag immutability at the registry level. Mutable tags in production are ticking time bombs.

2. docker pull is four HTTP requests: resolve tag to manifest, download config, download layers in parallel, verify SHA256 hashes. Knowing this helps you debug slow pulls, auth failures, and network issues.

3. Rate limits are real infrastructure constraints. Docker Hub's 100-pulls-per-6-hours limit for anonymous users breaks CI at scale. Mirror images into your own registry or use a pull-through cache.

4. Promote by digest, never by rebuild. Copying the exact bytes from dev to prod guarantees that what you tested is what you deploy. Rebuilding introduces drift.

5. Garbage collection is not automatic. Untagged manifests and orphaned blobs accumulate. Configure lifecycle policies (ECR) or schedule GC (Harbor, self-hosted registry) to control storage costs.

6. Registry scanning catches CVEs discovered after build time. Enable continuous scanning in your registry — build-time scanning alone misses vulnerabilities published between deploys.

---

Related Lessons

• What Happens When You docker build — follows the build side of the image lifecycle
• The Container Escape — security at the container runtime level
• GitOps: The Repo Is the Truth — where image references live in a GitOps workflow
• What Happens When You kubectl apply — what happens after the image reference reaches Kubernetes
• Kubernetes Services: How Traffic Finds Your Pod — the next layer after the image is pulled

Pages that link here

• Cross-Domain Lessons
• Gitops The Repo Is The Truth
• Kubernetes Services How Traffic Finds Your Pod
• The Container Escape
• What Happens When You Docker Build
• What Happens When You Kubectl Apply