Skip to content

Interview Gauntlet: Flaky CI Build

Category: Debugging Difficulty: L2-L3 Duration: 15-20 minutes Domains: CI/CD, Linux cgroups

Round 1: The Opening

Interviewer: "Your build passes locally every time but fails in CI about 30% of the time. The failures are in different tests each run — no consistent pattern. Where do you start?"

Strong Answer:

"Non-deterministic test failures that only happen in CI point to environmental differences between local and CI. The common categories: timing-dependent tests (race conditions, sleeps that are too short), resource-dependent tests (tests that need more CPU or memory than CI provides), ordering-dependent tests (tests that pass in isolation but fail when run after another test that leaves state), and network-dependent tests (tests that hit external services which are flaky or slow in CI). I'd start by looking at the failure logs across the last 10 failed builds to find patterns. Are the failing tests always in the same module (even if different tests)? Do they fail at the same phase (early, middle, late)? Are there timeout errors? Then I'd compare the CI environment with local: how many CPU cores, how much memory, is the CI runner shared with other jobs, what's the I/O speed (CI runners often use network-attached storage that's slower than local SSD). For the 30% failure rate specifically: that's high enough to be environmental, not a rare race condition. I'd run the test suite in CI with --verbose output including timestamps to see if there are unexpected pauses or slowdowns."

Common Weak Answers:

  • "The tests are just flaky — add retries." — Retries mask the root cause and increase build time. 30% failure rate with retries means builds take 1.3x-2x longer.
  • "Run tests with a fixed seed." — Good for ordering issues but doesn't address resource or timing problems.
  • "It works on my machine." — This is the attitude that allows flaky CI to persist. The CI environment is part of the system.

Round 2: The Probe

Interviewer: "You notice that the CI runner has 2 CPU cores and 4 GB RAM, while your local machine has 8 cores and 32 GB. But the tests aren't memory-intensive — they're CPU-intensive computation tests with assertions on execution time. Some tests assert that a function completes in under 100ms. In CI, these sometimes take 300-400ms. Is this a test problem or an infrastructure problem?"

What the interviewer is testing: Understanding of CPU resource allocation in containerized CI environments and why performance-based assertions are fragile.

Strong Answer:

"It's both. The tests are poorly designed if they assert on wall-clock execution time in a shared environment, but the infrastructure is also underprovisioned. In CI, the runner is likely a container or VM with 2 CPU cores, and those cores might be shared with other CI jobs via cgroup CPU shares or CFS (Completely Fair Scheduler) throttling. If the CI runner is a Kubernetes pod or a Docker container with --cpus=2, the CPU isn't 'reserved 2 full cores' — it's 'can use up to 2 cores worth of CPU time.' If other pods on the same node are competing for CPU, the CFS scheduler introduces throttling. I'd check for CFS throttling: inside the CI container, look at cat /sys/fs/cgroup/cpu/cpu.stat (cgroup v1) or cat /sys/fs/cgroup/cpu.stat (cgroup v2) and check nr_throttled and throttled_time. If throttled_time is high, the container is being CPU-throttled by the cgroup limit. The tests should be fixed to not assert on wall-clock time — use iteration counts, algorithmic assertions, or relative performance (this function should be faster than that function) instead. And the CI environment should have guaranteed CPU resources (set CPU requests = CPU limits in Kubernetes) to prevent noisy-neighbor throttling."

Trap Alert:

If the candidate bluffs here: The interviewer will ask "Where in the cgroup filesystem do you check for CPU throttling?" For cgroup v1: /sys/fs/cgroup/cpu,cpuacct/cpu.stat shows nr_periods, nr_throttled, and throttled_time. For cgroup v2: /sys/fs/cgroup/cpu.stat shows nr_throttled and throttled_usec. The numbers are cumulative — you need to sample twice and take the delta. It's fine to say "I'd check the cgroup cpu.stat file — I don't remember the exact fields but I know it tracks throttling counts."

Round 3: The Constraint

Interviewer: "You fix the time-assertion tests and set CPU requests equal to limits. Build stability improves to 95%. But 5% of builds still fail. The remaining failures are in integration tests that spin up a PostgreSQL test container using testcontainers. The container sometimes fails to start within the 30-second timeout. Why?"

Strong Answer:

"Testcontainers spins up Docker containers for integration tests, and the startup time depends on image pull speed, container creation overhead, and the time for the database to become ready. In CI, several things can make this slow. First, image caching: if the CI runner doesn't cache Docker images between runs, every build pulls the postgres:15 image from Docker Hub. That's ~150 MB and depends on the runner's internet bandwidth. CI runners in some cloud environments have rate-limited internet or route through a proxy. Second, Docker-in-Docker performance: if CI runs containers inside containers (common in Kubernetes-based CI), there's a performance penalty. Testcontainers running inside a DinD setup uses the host's storage driver twice (overlayfs on overlayfs), which is slow. Third, resource competition: the PostgreSQL container competes with the test runner for CPU and memory. If the cgroup limits are tight, PostgreSQL's startup (loading shared libraries, running initdb, creating template databases) takes longer. Fixes: use a local registry mirror or pre-cache the postgres image on the CI runner. Increase the testcontainers startup timeout to 60 seconds (it's just a timeout, not a delay). Or use a sidecar PostgreSQL service instead of testcontainers — in GitHub Actions, that's a services: block; in GitLab CI, it's a services: key. The sidecar starts before the test step and is ready when tests run."

The Senior Signal:

What separates a senior answer: Understanding the interaction between Docker-in-Docker, cgroup limits, and container startup performance. Most developers think of testcontainers as "just like running Docker locally" and don't account for the CI-specific overhead. Knowing about the overlayfs-on-overlayfs penalty and the Docker Hub rate limiting shows real CI infrastructure experience. Also: the pragmatic suggestion to use a CI-native service sidecar instead of testcontainers, which avoids the Docker-in-Docker problem entirely.

Round 4: The Curveball

Interviewer: "A developer proposes: 'Let's just mark flaky tests with @retry(3) and move on. We're wasting too much time on CI infrastructure.' What's your response?"

Strong Answer:

"I understand the frustration — fixing CI infrastructure isn't as visible as shipping features. But @retry(3) has hidden costs that compound over time. First, build duration: if 20 tests each have a 5% failure rate and each takes 30 seconds, adding 3 retries means on average one retry per build, adding 30 seconds. As more tests become flaky and get the retry annotation, build time creeps up. I've seen codebases where retries added 10 minutes to every build. Second, signal degradation: retries hide real bugs. If a new code change introduces a genuine 5% failure rate regression, it's invisible because the retry masks it. You find out when the failure rate hits 30% and retries can't save you. Third, normalization of failure: once the team accepts 'some tests are flaky,' the bar for new flakiness drops. Every test that 'sometimes fails' gets a retry annotation instead of investigation. Within 6 months, you have 50 retry-annotated tests and a CI pipeline that's slow and unreliable. My counter-proposal: fix the remaining 5% — it's container startup time, which has a known solution (CI service sidecars or image caching). That's a one-time investment. After that, adopt a zero-flaky policy: any test that fails more than once in 20 runs gets investigated and fixed or removed, not retried. Track flakiness metrics (failure rate per test over time) and treat it like production reliability."

Trap Question Variant:

The right answer is nuanced, not dogmatic. A candidate who says "retries are always wrong" is being purist. Retries for genuinely non-deterministic tests (like tests that depend on external services) can be appropriate with guardrails. The key is: retries should be rare, logged, and monitored, not the default response to flakiness. A candidate who says "retries are fine, just add them everywhere" doesn't understand the compounding cost.

Round 5: The Synthesis

Interviewer: "You've debugged CPU throttling, Docker-in-Docker overhead, and image pull speed. None of these are test code issues — they're infrastructure issues. How do you think about CI as infrastructure?"

Strong Answer:

"CI is production infrastructure for your development team. When CI is slow or flaky, developer productivity drops directly — engineers wait for builds, lose context during retries, and merge with less confidence. I'd treat CI with the same rigor as production: SLOs for build time (p95 build should complete in under 10 minutes), reliability targets (99% of builds that pass locally should pass in CI), and observability (metrics on build duration, failure rate, queue wait time, and flakiness per test). Concretely: the CI infrastructure should be provisioned with dedicated resources, not shared burstable instances. Docker images should be cached locally (registry mirror, or bake common images into the runner's base image). The cgroup limits should be tested and tuned — set CPU requests equal to limits to avoid throttling, and give integration tests enough memory for their dependencies. Monitor the CI pipeline the same way you monitor production: if build times regress by 20%, an alert fires and someone investigates. And review CI infrastructure the same way you review application architecture — periodically audit: are we using the right runner size? Are our caching strategies effective? Are there tests that should be parallelized or moved to a separate stage? The teams that ship fastest are the ones that invest in their CI infrastructure as a first-class concern, not an afterthought."

What This Sequence Tested:

Round Skill Tested
1 Systematic flaky test investigation methodology
2 Linux cgroup CPU throttling mechanics in containerized environments
3 Docker-in-Docker performance and CI-specific container issues
4 Engineering judgment on technical debt trade-offs
5 CI infrastructure strategy and developer productivity thinking

Prerequisite Topic Packs