What Happens When You `docker build`

lesson
dockerfile
layers
overlay-filesystem
build-cache
registries
multi-stage-builds
image-security ---# What Happens When You docker build

Topics: Dockerfile, layers, overlay filesystem, build cache, registries, multi-stage builds, image security Level: L1–L2 (Foundations → Operations) Time: 60–90 minutes Prerequisites: None (everything is explained from scratch)

The Mission¶

You type docker build -t myapp:v1 . and press Enter. Twenty seconds later, you have an image. But what actually happened? Where did 20 seconds go? Why is the image 800MB when your app is 15MB? Why does changing one line of code rebuild everything? And where does the image actually live?

This lesson follows docker build from the moment you press Enter through every layer of the build process: context sending, instruction parsing, layer creation, caching, the union filesystem, and all the way to the registry. By the end, you'll understand Docker images well enough to make them fast, small, and secure.

Step 1: Sending the Build Context¶

The first thing Docker does is send the build context to the Docker daemon. The build context is everything in the directory you specified (. in our command).

Sending build context to Docker daemon  245.8MB

245MB? Your app is 15MB. What's in there?

# See what's being sent
du -sh .git/ node_modules/ .venv/ *.log
# → 180MB  .git/
# → 45MB   node_modules/
# → 15MB   .venv/
# → 5MB    app.log

The daemon received everything — your Git history, your dependencies, your log files, maybe even your .env with secrets. This is why .dockerignore exists:

# .dockerignore
.git
node_modules
.venv
*.log
.env
__pycache__
.pytest_cache

With .dockerignore, the context drops to 15MB and the build starts instantly.

Gotcha: Without .dockerignore, COPY . . sends everything to the daemon AND copies it into the image layer. Your .git directory, your node_modules, your .env with production credentials — all baked into the image, visible to anyone who pulls it. Even if you RUN rm .env later, the file exists in an earlier layer and can be extracted. Docker images are not like zip files — every layer is independently readable.

Under the Hood: docker build is a client-server operation. The CLI (docker) sends the build context to the daemon (dockerd) over a Unix socket or TCP. This is why Docker Desktop on macOS is slow for large contexts — the context has to cross the VM boundary. BuildKit (the modern builder) streams files on-demand instead of sending everything upfront, which is much faster for large repos.

Step 2: Parsing the Dockerfile¶

The daemon reads your Dockerfile instruction by instruction. Each instruction does one of two things:

Instruction	Creates a layer?	What it does
`FROM`	Yes (base image)	Sets the base image — all layers below this are inherited
`RUN`	Yes	Executes a command and saves the filesystem diff
`COPY` / `ADD`	Yes	Adds files from the build context
`ENV`	No (metadata)	Sets an environment variable in the image config
`EXPOSE`	No (metadata)	Documents a port (does nothing for networking)
`LABEL`	No (metadata)	Adds key-value metadata
`WORKDIR`	No (metadata)	Sets the working directory
`CMD` / `ENTRYPOINT`	No (metadata)	Defines what runs when the container starts
`USER`	No (metadata)	Sets the user for subsequent instructions

Remember: The FRAC mnemonic — FROM, RUN, ADD/COPY create layers. Everything else is metadata stored in the image config.

Step 3: Building Layers — What Actually Happens¶

For each layer-creating instruction, Docker:

Creates a temporary container from the previous layer
Executes the instruction inside that container
Takes a snapshot of the filesystem diff (what changed)
Saves the diff as a new layer
Destroys the temporary container

Let's trace through a real Dockerfile:

FROM python:3.11-slim                    # Layer 1: base image (50+ layers from upstream)

WORKDIR /app                             # Metadata only — no new layer

COPY requirements.txt .                  # Layer 2: one file added
RUN pip install --no-cache-dir -r requirements.txt  # Layer 3: packages installed

COPY . .                                 # Layer 4: app code added

USER 1000                                # Metadata only
EXPOSE 8000                              # Metadata only
CMD ["uvicorn", "app:app", "--host", "0.0.0.0"]  # Metadata only

# See the layers after building
docker history myapp:v1
# IMAGE          CREATED        CREATED BY                              SIZE
# abc123         5 seconds ago  CMD ["uvicorn" "app:app" "--host" ...   0B
# def456         5 seconds ago  COPY . .                                15MB
# 789abc         8 seconds ago  RUN pip install --no-cache-dir -r ...   45MB
# fedcba         9 seconds ago  COPY requirements.txt .                 1KB
# ...            2 weeks ago    (base image layers)                     ...

Under the Hood: Each layer is a tar archive of the filesystem diff — files that were added, modified, or deleted relative to the previous layer. Layers are identified by their content hash (SHA256 digest), not by the instruction that created them. Two identical COPY operations on the same files produce the same layer hash — this is content- addressable storage, the same idea behind Git's object model.

Step 4: The Build Cache — Why Order Matters¶

Docker caches each layer by its instruction + inputs. If neither changed, the cache is reused:

Step 2/6 : COPY requirements.txt .
 ---> Using cache                     ← Cache hit! Reused from last build.

Step 3/6 : RUN pip install ...
 ---> Using cache                     ← Also cached — input (requirements.txt) didn't change.

Step 4/6 : COPY . .
 ---> abc123def                       ← Cache miss — app code changed.

Critical rule: Once a layer's cache is invalidated, ALL subsequent layers must be rebuilt. This is why instruction order matters enormously:

# BAD — changing any source file busts the pip install cache
COPY . .                                    # ← changes every commit
RUN pip install -r requirements.txt         # ← rebuilt every time!

# GOOD — dependencies cached separately from code
COPY requirements.txt .                     # ← changes only when deps change
RUN pip install -r requirements.txt         # ← cached until deps change
COPY . .                                    # ← code changes only rebuild this layer

The good version rebuilds pip install only when requirements.txt changes. The bad version rebuilds it on every code change. On a project with 200 dependencies, that's the difference between a 5-second build and a 5-minute build.

Mental Model: Think of layers like a stack of transparent sheets. Each sheet is glued to the one below. If you change sheet 3, you have to redo sheets 3, 4, 5, 6... even if 4, 5, 6 didn't change. So put the things that change least often at the bottom (OS, dependencies) and the things that change most often at the top (your code).

Step 5: The Union Filesystem — How Layers Become a Filesystem¶

An image is a stack of read-only layers. A container adds one writable layer on top. The union filesystem (OverlayFS on modern Linux) merges them into a single coherent view:

┌─────────────────────────┐
│    Container layer       │  ← writable (upperdir)
│    (your runtime changes)│
├─────────────────────────┤
│    Layer 4: COPY . .     │  ← read-only
├─────────────────────────┤
│    Layer 3: pip install  │  ← read-only
├─────────────────────────┤
│    Layer 2: requirements │  ← read-only
├─────────────────────────┤
│    Layer 1: python:slim  │  ← read-only (many sub-layers)
└─────────────────────────┘
         merged view = what the container sees as /

# See how OverlayFS is mounted for a running container
mount | grep overlay
# → overlay on /var/lib/docker/overlay2/.../merged type overlay
#   (lowerdir=...:...:..., upperdir=..., workdir=...)

OverlayFS term	What it is
`lowerdir`	Image layers (read-only, stacked)
`upperdir`	Container's writable layer
`workdir`	Overlay bookkeeping (internal use)
`merged`	What the container sees as its root filesystem

Copy-on-write¶

When a container modifies a file from a lower (read-only) layer, OverlayFS copies the entire file to the writable upper layer before applying the change. This is copy-on-write (COW).

Implications: - Reading files from lower layers is fast (no copy needed) - Writing a file for the first time is slow (full file copied up, then modified) - Large files modified in the container (databases, logs) are copied entirely - Deleted files aren't really deleted — a whiteout marker in the upper layer hides them

Gotcha: If you RUN rm -rf /large-directory in a Dockerfile, the files are still in the previous layer. The rm creates whiteout markers in a new layer, which hides the files from the container's view — but anyone who inspects the image layers can still see (and extract) the original files. This is why credentials accidentally added in an early layer can't be "removed" by deleting them later. Use multi-stage builds to avoid this.

Step 6: Multi-Stage Builds — Small, Clean Images¶

The biggest win in Dockerfile design: separate build tools from the runtime image.

# Stage 1: Build (fat, has compilers and build tools)
FROM golang:1.22 AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /server

# Stage 2: Runtime (tiny, has only the binary)
FROM gcr.io/distroless/static:nonroot
COPY --from=builder /server /server
USER 65534
ENTRYPOINT ["/server"]

Build stage: 1.2GB (Go toolchain, source code, intermediate objects). Runtime stage: 15MB (static binary + minimal base).

The build tools, source code, and intermediate files exist only in the builder stage — they never appear in the final image. This is why multi-stage builds are the single most important Dockerfile technique.

Trivia: Multi-stage builds were introduced in Docker 17.05 (2017). Before that, people used shell scripts that ran docker build twice and docker cp to extract artifacts. The multi-stage feature was one of the most requested Docker features ever.

Base image choices¶

Base	Size	Has shell?	Has package manager?	Best for
Ubuntu	~75MB	Yes (bash)	Yes (apt)	When you need a full environment
Debian slim	~75MB	Yes (bash)	Yes (apt)	Smaller but still general purpose
Alpine	~7MB	Yes (ash)	Yes (apk)	Small images, but musl libc has gotchas
Distroless	~20MB	No	No	Maximum security, can't debug without ephemeral containers
Scratch	0MB	No	No	Static binaries only

Gotcha: Alpine uses musl libc instead of glibc. This breaks some Python C extensions, causes different DNS resolution behavior, and can have thread performance differences. If your Python app needs numpy, pandas, or grpcio, use python:3.11-slim (Debian + glibc) instead of python:3.11-alpine. The build time saved by Alpine is lost to compiling C extensions from source.

Step 7: Pushing to a Registry¶

The image exists locally. To share it, push it to a registry:

# Tag for a registry
docker tag myapp:v1 registry.example.com/myapp:v1

# Push
docker push registry.example.com/myapp:v1

What happens during push:

Docker reads the image manifest (list of layer digests + config)
For each layer, checks if the registry already has it (by digest)
Uploads only the layers the registry doesn't have
Uploads the config object
Uploads the manifest (which ties everything together)

Because layers are content-addressed, two images that share base layers only upload the difference. Pushing myapp:v2 when only the top layer changed uploads 15MB, not 800MB.

# See image manifest
docker inspect --format='{{.RootFS.Layers}}' myapp:v1
# → [sha256:abc123... sha256:def456... sha256:789abc... sha256:fedcba...]

Tags vs digests¶

# Tag — mutable, can point to different images over time
docker pull myapp:v1

# Digest — immutable, content-addressed, guaranteed to be the same image forever
docker pull myapp@sha256:abc123def456...

Gotcha: The :latest tag is not special — it's just a tag like any other, and it's mutable. docker pull myapp:latest can return a different image every time. In production, always pin by digest or use explicit version tags. :latest in a Kubernetes Deployment means a pod restart can silently change the running code.

Step 8: Security — What's in Your Image?¶

Every layer of your image is a potential attack surface. Scan it:

# Scan for known vulnerabilities
trivy image myapp:v1
# → Total: 15 (UNKNOWN: 0, LOW: 8, MEDIUM: 5, HIGH: 2, CRITICAL: 0)

# Only show high and critical
trivy image --severity HIGH,CRITICAL myapp:v1

Security checklist for every Dockerfile:

# 1. Non-root user
USER 1000:1000

# 2. No secrets in the image (use build secrets instead)
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm install

# 3. Read-only filesystem at runtime
# (set in docker run or K8s securityContext, not Dockerfile)

# 4. Minimal base image (less code = fewer vulnerabilities)
FROM gcr.io/distroless/static:nonroot

# 5. Pin base image by digest for reproducibility
FROM python:3.11-slim@sha256:abc123...

War Story: A Go service worked perfectly in dev but crashed in production with a cryptic error. Same image SHA everywhere. Root cause: production had a custom seccomp profile that blocked mmap with PROT_EXEC flags. Dev used the default (permissive) profile. "Same image" does NOT mean "same runtime environment" — security policies, syscall filters, and resource limits differ between environments.

The Complete Build — One Picture¶

[1] docker build -t myapp:v1 .
    CLI sends build context to daemon (minus .dockerignore)

[2] Daemon reads Dockerfile
    Parses instructions: FROM, RUN, COPY create layers; rest is metadata

[3] For each layer-creating instruction:
    Create temp container → execute → snapshot diff → save as layer → destroy

[4] Build cache:
    Same instruction + same inputs = reuse cached layer
    Cache bust → rebuild this layer AND everything after

[5] Result: ordered stack of layer tarballs + config JSON
    Content-addressed by SHA256 digest

[6] Tag: myapp:v1 → pointer to manifest

[7] Push: upload missing layers + manifest to registry
    Shared layers (base images) are not re-uploaded

[8] Pull: resolve tag/digest → fetch manifest → download missing layers → unpack

[9] Run: stack layers (read-only) + writable top layer via OverlayFS → merged /

Flashcard Check¶

Q1: What does .dockerignore prevent?

Prevents files from being sent to the daemon as build context AND from being included in COPY/ADD layers. Critical for excluding .git, node_modules, .env.

Q2: RUN rm -rf /tmp/secrets — are the secrets gone from the image?

No. They exist in a previous layer. The rm creates whiteout markers in a new layer that hide the files, but the previous layer is still extractable. Use multi-stage builds to avoid putting secrets in the image at all.

Q3: Why does changing one line of code rebuild the pip install layer?

Because COPY . . (which includes the changed file) comes before RUN pip install. Cache invalidation on COPY busts all subsequent layers. Fix: copy dependency files first, install deps, then copy code.

Q4: What's the FRAC mnemonic?

FROM, RUN, ADD/COPY — these create layers. Everything else (ENV, EXPOSE, LABEL, CMD, USER, WORKDIR) is metadata stored in the image config.

Q5: Alpine images are 7MB vs 75MB for Debian slim. Should you always use Alpine?

No. Alpine uses musl libc, which breaks some Python C extensions, has different DNS behavior, and can have thread performance issues. Use slim for Python/Ruby apps that need C extensions.

Q6: docker pull myapp:latest — is this reproducible?

No. :latest is a mutable tag that can point to different images over time. Use explicit version tags or pin by digest (@sha256:...) for reproducibility.

Exercises¶

Exercise 1: Optimize a Dockerfile (refactor)¶

This Dockerfile works but is slow and produces a 900MB image:

FROM python:3.11
COPY . .
RUN pip install -r requirements.txt
RUN apt-get update && apt-get install -y curl
EXPOSE 8000
CMD ["python", "app.py"]

Rewrite it to be fast (good cache behavior) and small (minimal image).

Solution

FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt
COPY . .

FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY --from=builder /app .
ENV PATH=/root/.local/bin:$PATH
RUN useradd -r -u 1000 appuser
USER appuser
EXPOSE 8000
CMD ["python", "app.py"]

Changes: slim base (75MB vs 900MB), multi-stage build, dependency-first caching, non-root user, no curl in runtime image.

Exercise 2: Find the secret (investigation)¶

Build this Dockerfile:

FROM alpine
RUN echo "API_KEY=sk-secret123" > /tmp/creds.txt
RUN rm /tmp/creds.txt
CMD ["sh"]

The secret was "deleted." Can you find it?

How to find it

docker build -t secret-test .
docker history secret-test
# Note the layer ID for the RUN echo step

docker save secret-test | tar -xf - -O */layer.tar | tar -tf - 2>/dev/null | grep creds
# Or use dive: dive secret-test

The `echo` instruction created a layer containing the file. The `rm` created a new layer with a whiteout marker. The file is still in the earlier layer.

Exercise 3: Measure cache impact (hands-on)¶

# Build with cache cold
docker build --no-cache -t myapp:test .
# Note the time

# Change one source file, rebuild
touch app.py
docker build -t myapp:test .
# Note the time and which steps were cached

# Now move COPY . . before RUN pip install, rebuild
# Compare the times

Exercise 4: Inspect layers (hands-on)¶

# See layer sizes
docker history myapp:v1

# See layer digests
docker inspect --format='{{range .RootFS.Layers}}{{.}}{{"\n"}}{{end}}' myapp:v1

# See total image size vs sum of layers
docker images myapp:v1
docker system df -v | grep myapp

Which layer is the biggest? Could you make it smaller?

Cheat Sheet¶

Build Commands¶

Task	Command
Build with tag	`docker build -t myapp:v1 .`
Build without cache	`docker build --no-cache -t myapp:v1 .`
Multi-platform	`docker buildx build --platform linux/amd64,linux/arm64 -t reg/myapp:v1 --push .`

Debugging Images¶

Task	Command
Layer sizes	`docker history myapp:v1`
Layer digests	`docker inspect --format='{{.RootFS.Layers}}' myapp:v1`
Image config	`docker inspect myapp:v1`
Disk usage	`docker system df -v`
Run as shell	`docker run --rm -it --entrypoint sh myapp:v1`

Dockerfile Best Practices¶

# 1. Specific base (not :latest)
FROM python:3.11-slim

# 2. Dependencies before code (cache optimization)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# 3. Code last (changes most often)
COPY . .

# 4. Non-root user
USER 1000

# 5. .dockerignore (exclude .git, node_modules, .env, __pycache__)

Layer Rule¶

FROM, RUN, ADD, COPY → create layers (FRAC)
Everything else → metadata only (zero bytes)

Takeaways¶

.dockerignore is not optional. Without it, you're sending .git, credentials, and gigabytes of junk to the daemon and baking it into every image.
Layer order determines build speed. Dependencies before code. Things that change least at the bottom, most at the top. One wrong COPY placement can turn a 5-second build into a 5-minute build.
Layers are permanent. Deleting a file in a later layer doesn't remove it from earlier layers. Use multi-stage builds to keep secrets and build tools out of the final image.
Multi-stage builds cut image size 10-50x. Build tools stay in the builder stage. Only runtime artifacts reach the final image.
Tags are mutable, digests are immutable. :latest can change silently. Pin by version tag or digest for production reproducibility.
Scan your images. trivy image myapp:v1 finds known vulnerabilities. Run it in CI before pushing.

The Hanging Deploy — what happens when the container starts running
What Happens When You Press Power — the boot sequence that Docker skips
Permission Denied — container security contexts and non-root users