Skip to content

The Container That Worked on My Machine

Category: The Mystery Domains: containers, linux-ops Read time: ~5 min

Setting the Scene

We had a Go service that processed PDF reports -- nothing exotic, just a wrapper around a C library via cgo. The Docker image was identical everywhere: same SHA, same registry, same docker inspect output. It worked flawlessly in dev, passed all CI tests, and crashed hard within 30 seconds of hitting production. Every single time.

The error was unhelpful: unexpected fault address. No stack trace, no core dump, just a dead process. I was the one who'd written the service, so naturally I was the one who got paged at midnight.

What Happened

My first instinct was a config difference. I diffed every environment variable, every mounted secret, every ConfigMap. Identical. I pulled the exact image SHA from the prod registry, ran it on my laptop with the same env vars. Worked perfectly. I ran it on a staging box that mirrored prod hardware. Worked perfectly.

I spent the next day convinced it was a kernel version difference. Prod was running 5.15, my laptop was on 5.19, staging was on 5.15 too. But staging worked. I started reading kernel changelogs, looking for syscall behavior changes between minor versions. Dead end.

Day three, I added more logging. I compiled with -race, added signal handlers, wrapped every C call with recovery. The crash still happened, but now I got slightly more context: it was dying inside a mmap call in the C library. The library was trying to map memory with PROT_EXEC -- executable memory pages.

A coworker who'd worked in security asked: "What seccomp profile is prod running?" I hadn't even thought about it. Our dev environments used Docker's default seccomp profile. Our production Kubernetes cluster used a custom, hardened profile that the security team had rolled out two months earlier.

I ran kubectl get pod -o json | jq '.spec.containers[].securityContext' on a prod pod and found a custom seccomp profile at /var/lib/kubelet/seccomp/hardened.json. I compared it to Docker's default using diff. The custom profile blocked mmap with PROT_EXEC flags -- exactly what the C library needed to JIT-compile PDF rendering instructions.

The Moment of Truth

I ran strace -f -e trace=mmap on the container in both environments. In dev, the mmap call with PROT_READ|PROT_WRITE|PROT_EXEC succeeded. In prod, it returned EPERM. The seccomp filter was silently killing the syscall, and the C library didn't handle the error gracefully -- it just segfaulted.

The Aftermath

We worked with the security team to add a narrow exception for mmap with exec permissions in the seccomp profile, scoped only to that service's pods. Long-term, we replaced the C library with a pure-Go alternative that didn't need executable memory. The whole ordeal took four days and taught me that "same image" does not mean "same runtime environment."

The Lessons

  1. Security profiles differ between environments: seccomp, AppArmor, and SELinux can silently change syscall behavior. Always compare the full security context, not just the image.
  2. Test with production security policies: Your CI pipeline should run integration tests under the same seccomp and AppArmor profiles as production.
  3. strace is your friend: When a process dies with no useful error, strace -f will show you exactly which syscall failed and why. It's the fastest path to truth.

What I'd Do Differently

Add a CI stage that runs the container under the production seccomp profile. Maintain a diff report between dev and prod security contexts as part of the deploy pipeline. And never again assume "same image = same behavior."

The Quote

"Same image, same config, same everything -- except the invisible security policy that was silently killing my syscalls."

Cross-References