Skip to content

Ops Archaeology: The Container That Exits Immediately

You've just joined a team. There are no docs. The previous engineer left last month. Something is broken. Here's everything you have to work with.

Difficulty: L1 Estimated time: 15 min Domains: Docker, Container Images, CI/CD


Artifact 1: CLI Output

$ docker ps -a --filter name=pdf-renderer
CONTAINER ID   IMAGE                              COMMAND              CREATED          STATUS                      PORTS   NAMES
a3f8b2c1d4e5   registry.corp.io/pdf-renderer:v3   "/app/render"        12 seconds ago   Exited (1) 11 seconds ago           pdf-renderer
9c7e6d5f4a3b   registry.corp.io/pdf-renderer:v3   "/app/render"        47 seconds ago   Exited (1) 46 seconds ago           pdf-renderer
f1b2c3d4e5a6   registry.corp.io/pdf-renderer:v3   "/app/render"        1 minute ago     Exited (1) 1 minute ago             pdf-renderer
e5d4c3b2a1f0   registry.corp.io/pdf-renderer:v3   "/app/render"        2 minutes ago    Exited (1) 2 minutes ago            pdf-renderer

$ docker logs a3f8b2c1d4e5
exec /app/render: exec format error

$ docker inspect a3f8b2c1d4e5 --format '{{.Config.Image}} | Architecture: {{.Architecture}} | OS: {{.Os}}'
registry.corp.io/pdf-renderer:v3 | Architecture: arm64 | OS: linux

$ uname -m
x86_64

Artifact 2: Metrics

# No Prometheus metrics available — the pdf-renderer container never starts
# long enough to serve its /metrics endpoint.

# Docker daemon metrics from the host:
engine_daemon_container_states_containers{state="stopped"} 47
engine_daemon_container_actions_seconds_count{action="start"} 312

# Upstream service queue depth (from a different exporter):
pdf_queue_pending_jobs 1847
pdf_queue_oldest_job_age_seconds 7234

Artifact 3: Infrastructure Code

# From: Dockerfile (last modified 3 days ago in git)
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --production
COPY . .
RUN npm run build

FROM node:20-alpine
WORKDIR /app
COPY --from=builder /app/dist ./
COPY --from=builder /app/node_modules ./node_modules
ENTRYPOINT ["/app/render"]

Artifact 4: Log Lines

[2024-11-15T09:12:03Z] docker     | exec /app/render: exec format error
[2024-11-15T09:11:58Z] ci-runner  | Step 8/8 : Successfully built amd64/linux image pdf-renderer:v3 on runner gitlab-runner-arm64-02
[2024-11-15T09:10:44Z] registry   | PUT /v2/pdf-renderer/manifests/v3 digest=sha256:8a3b... size=1842 arch=arm64

Your Mission

  1. Reconstruct: What does this system do? What are its components and purpose?
  2. Diagnose: What is currently broken or degraded, and why?
  3. Propose: What would you do to fix it? What would you check first?