The Cascading Timeout

lesson
circuit-breakers
retry-storms
backpressure
thread-pools
timeouts
distributed-failure
l2 ---# The Cascading Timeout

Topics: circuit breakers, retry storms, backpressure, thread pools, timeouts, distributed failure Level: L2 (Operations) Time: 60–75 minutes Prerequisites: Basic understanding of microservices helpful but not required

The Mission¶

One backend service gets slow. Within 5 minutes, every service in your platform is returning errors. Users see a blank page. Your dashboard is a wall of red. The cascading failure has begun.

This lesson traces how a single slow dependency can bring down an entire distributed system, and teaches you the patterns that prevent it: circuit breakers, timeouts, backpressure, bulkheads, and retry discipline.

The Anatomy of a Cascade¶

Minute 0:  Database slow (disk contention from backup)
           → Payment service queries take 10s instead of 50ms

Minute 1:  Payment service thread pool fills up (all threads waiting on DB)
           → New requests queue behind blocked threads
           → Payment service stops responding to health checks

Minute 2:  Order service (calls Payment) times out after 30s default
           → But each timeout consumes a thread in Order service for 30s
           → Order service thread pool fills up too

Minute 3:  API Gateway (calls Order) times out
           → Gateway thread pool fills up
           → Gateway stops accepting new connections
           → Users see 504 Gateway Timeout

Minute 4:  Users refresh the page (retry)
           → Each refresh creates a NEW request through the entire chain
           → 3x the traffic hitting an already-broken system

Minute 5:  Everything is down. The database backup finished 2 minutes ago.
           The DB is fine now. But the cascade is self-sustaining.

Mental Model: A cascade is like a traffic jam. One car brakes, the car behind brakes harder, the car behind that brakes harder still. Soon, traffic is stopped a mile back from the original cause — which has long since cleared. The cascade is self-sustaining even after the trigger resolves.

Pattern 1: Timeouts — Stop Waiting Forever¶

The default HTTP client timeout in many frameworks is infinity or 30-60 seconds. A service waiting 30 seconds for a response is consuming a thread, a connection, and memory for 30 seconds — multiplied by every concurrent request.

# BAD — default timeout (could be 30s, could be infinite)
response = requests.get("http://payment-service/api/charge")

# GOOD — explicit timeout: 2s connect, 5s read
response = requests.get(
    "http://payment-service/api/charge",
    timeout=(2, 5)
)

Timeout budget¶

If your SLA is 500ms for the user-facing request, and you call 3 services in sequence:

User SLA: 500ms

API Gateway:    50ms overhead
  → Auth service:    100ms timeout
  → Order service:   200ms timeout
    → Payment service: 100ms timeout
Total budget:         450ms (50ms headroom)

If Payment gets a 30-second timeout, a single slow Payment response blows through every timeout in the chain.

Gotcha: Timeout = connect timeout + read timeout. A 5-second timeout on requests.get means 5 seconds TOTAL — including DNS, TCP handshake, TLS, and waiting for response. If DNS takes 3 seconds (ndots problem!), you only have 2 seconds for the actual response. Always set connect and read timeouts separately.

Pattern 2: Circuit Breakers — Stop Calling Broken Services¶

A circuit breaker monitors the failure rate of calls to a dependency. When failures exceed a threshold, it "opens" and immediately returns an error — without making the call.

State: CLOSED (normal)
  ↓ Failure rate > 50% for 10 seconds
State: OPEN (rejecting all calls immediately)
  ↓ After 30 seconds
State: HALF-OPEN (allow one probe request through)
  ↓ Probe succeeds → CLOSED (resume normal traffic)
  ↓ Probe fails → OPEN (wait another 30 seconds)

# Python with tenacity or similar
from circuitbreaker import circuit

@circuit(failure_threshold=5, recovery_timeout=30)
def call_payment_service(order_id):
    response = requests.post(
        "http://payment-service/api/charge",
        json={"order_id": order_id},
        timeout=(2, 5)
    )
    response.raise_for_status()
    return response.json()

# When circuit is OPEN:
# → Immediately raises CircuitBreakerError
# → No network call, no thread blocked, no timeout wait
# → Return graceful degradation to user instead

Name Origin: Circuit breakers in software are named after electrical circuit breakers — devices that "trip" when current exceeds a safe level, disconnecting the circuit to prevent damage. The pattern was popularized by Michael Nygard's Release It! (2007) and implemented in Netflix's Hystrix library (2012). Netflix needed them because a single slow microservice could cascade and take down all of Netflix.

Without a circuit breaker, every request to a broken service ties up a thread for the full timeout duration. With one, failing requests return immediately — the calling service stays healthy even when the dependency is dead.

Pattern 3: Retry Storms — The Amplification Problem¶

When a service fails, clients retry. Each retry is a new request. If 1,000 clients each retry 3 times, a service that was handling 1,000 requests/second now receives 4,000.

Normal load: 1,000 req/s → Service handles fine
Service hiccups: drops 10% → 100 failures
Clients retry 3x each: 100 × 3 = 300 extra requests
New load: 1,300 req/s → Maybe fine

Service partially down: drops 50% → 500 failures
Clients retry 3x each: 500 × 3 = 1,500 extra requests
New load: 2,500 req/s → Service collapses completely
All 2,500 fail → clients retry again → 10,000 req/s → death spiral

War Story: An e-commerce platform's payment service hit a database lock during a flash sale. Response times spiked from 200ms to 30 seconds. The mobile app retried failed payment attempts up to 5 times with no backoff. Each retry re-acquired the same database lock. The retry storm amplified the original problem by 5x, locked the database entirely, and cascaded through the order service, inventory service, and API gateway. The entire platform was down for 23 minutes — 18 minutes after the original database lock resolved.

The fix: Exponential backoff + jitter¶

import random
import time

def retry_with_backoff(func, max_retries=3):
    for attempt in range(max_retries):
        try:
            return func()
        except Exception:
            if attempt == max_retries - 1:
                raise
            # Exponential backoff with jitter
            delay = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(delay)
            # Attempt 0: ~1s, Attempt 1: ~2-3s, Attempt 2: ~4-5s

The jitter is critical. Without it, all clients that failed at the same time retry at the same time — creating synchronized spikes. Random jitter spreads retries across time.

Pattern 4: Bulkheads — Isolate Failures¶

Named after the watertight compartments in a ship's hull: if one compartment floods, the others stay dry. In software, bulkheads isolate resources per dependency:

# BAD — one thread pool shared across all dependencies
# If payment service is slow, it consumes all threads
# Now auth and inventory are also blocked (no threads available)

# GOOD — separate thread pool per dependency
payment_pool = ThreadPoolExecutor(max_workers=10)
auth_pool = ThreadPoolExecutor(max_workers=5)
inventory_pool = ThreadPoolExecutor(max_workers=10)

# Payment service being slow only consumes payment_pool threads
# Auth and inventory have their own pools and keep working

Kubernetes does this at the infrastructure level with resource limits — one pod's resource consumption can't starve others on the same node.

Pattern 5: Backpressure — Slow Down Instead of Falling Over¶

When a system is overwhelmed, it has two choices: accept everything and crash, or push back and tell callers to slow down.

HTTP 429 Too Many Requests:

HTTP/1.1 429 Too Many Requests
Retry-After: 5

Kubernetes resource requests are a form of backpressure — the scheduler won't place a pod on a node that can't handle it.

Message queue consumer lag is a form of backpressure — the queue holds messages until the consumer can process them, rather than dropping or crashing.

Mental Model: Backpressure is like water flow. A pipe that's too small doesn't explode — water backs up. A system that accepts more work than it can process DOES explode (OOM, thread exhaustion, timeout cascade). Explicit backpressure (429, queue depth limits, connection limits) is the pipe that safely backs up instead of bursting.

The Complete Defense Stack¶

            ┌─────────────────────┐
User ─────→│  API Gateway         │
            │  • Rate limiting     │  ← Backpressure (429)
            │  • Connection limit  │
            └──────────┬──────────┘
                       │
            ┌──────────▼──────────┐
            │  Service A           │
            │  • Timeout: 2s       │  ← Stop waiting
            │  • Circuit breaker   │  ← Stop calling broken deps
            │  • Bulkhead pools    │  ← Isolate per dependency
            │  • Retry: 2x + jitter│  ← Don't amplify failure
            └──────────┬──────────┘
                       │
            ┌──────────▼──────────┐
            │  Service B (slow)    │
            │  • Health check      │  ← Report "not ready" honestly
            │  • Graceful degrade  │  ← Return cached/default response
            └─────────────────────┘

Every layer defends itself: 1. Gateway: rate limits and connection limits (backpressure) 2. Each service: timeout + circuit breaker + bulkhead (isolation) 3. Retries: exponential backoff + jitter + max attempts (discipline) 4. Health checks: remove unhealthy instances from rotation (honest reporting)

Flashcard Check¶

Q1: A slow database causes all services to fail. What pattern prevents this?

Circuit breaker. When the failure rate exceeds a threshold, the breaker opens and returns errors immediately without calling the slow service. This keeps upstream services healthy.

Q2: 1,000 clients each retry 3 times. What's the effective load?

Up to 4,000 requests/second (original 1,000 + 3,000 retries). With synchronized retries, the load arrives in spikes. Add exponential backoff + jitter.

Q3: What's the difference between a timeout and a circuit breaker?

Timeout: stops waiting for ONE slow request. Circuit breaker: stops making ANY requests to a dependency that's failing. Timeout prevents one bad call from blocking; circuit breaker prevents the entire pattern of bad calls.

Q4: What is a bulkhead in software?

Separate resource pools per dependency. If the payment thread pool is exhausted, the auth and inventory pools are unaffected. Named after ship hull compartments.

Q5: Why is jitter important in retries?

Without jitter, all clients that failed at time T retry at time T+1. This creates synchronized spikes. Random jitter spreads retries across time, smoothing the load.

Exercises¶

Exercise 1: Design timeout budgets (think)¶

Your user-facing endpoint has a 1-second SLA. It calls Auth (fast), then Inventory (fast), then Payment (sometimes slow). Design the timeout budget.

One approach

Total SLA: 1000ms
Gateway overhead:    50ms
Auth service:       100ms timeout (typically 20ms)
Inventory service:  200ms timeout (typically 50ms)
Payment service:    500ms timeout (typically 100ms)
Headroom:           150ms

Key insight: Payment gets the biggest timeout because it has the most variance. But 500ms is still far less than the default 30 seconds, which would blow through the entire SLA.

Exercise 2: The decision (think)¶

For each scenario, which pattern helps?

External API is down — every call hangs for 30 seconds
Flash sale causes 10x normal traffic
One slow database query blocks all API requests
Recovering service gets hit with thundering herd of retries
User keeps clicking "submit" while the page loads

Answers

1. **Circuit breaker + timeout.** Circuit breaker stops calling after N failures. Timeout limits the damage of each individual call. 2. **Rate limiting (backpressure).** Return 429 to excess traffic. Auto-scaling helps but takes minutes; rate limiting is immediate. 3. **Bulkhead.** Separate thread/connection pools per dependency. The slow database query doesn't consume all threads — only its pool. 4. **Exponential backoff + jitter.** Spread retries across time. Also consider circuit breaker on the client side. 5. **Idempotency + deduplication.** Each click creates the same order ID. Server processes it once, returns cached result for duplicates.

Cheat Sheet¶

Pattern	What it does	When to use
Timeout	Stops waiting for slow responses	Every external call. Always.
Circuit breaker	Stops calling failing dependencies	Any non-trivial dependency
Retry + backoff + jitter	Retries failures without amplifying	Transient errors only (not 400s)
Bulkhead	Isolates resources per dependency	When one slow dep mustn't kill others
Backpressure	Rejects excess load gracefully	When you know your capacity limits
Rate limiting	Caps request rate per client/total	APIs with external consumers
Graceful degradation	Returns cached/default when dep fails	When partial response > no response

Takeaways¶

Timeouts are mandatory. The default timeout is often 30s or infinity. Set explicit connect + read timeouts on every external call. Every one.
Circuit breakers prevent cascades. A slow dependency ties up threads for the full timeout. A circuit breaker returns immediately, keeping your service alive.
Retries without backoff are DDoS. Exponential backoff + jitter turns a retry storm into a gentle recovery. Without jitter, retries are synchronized spikes.
Bulkheads contain damage. One slow dependency should only fill its own thread pool, not the shared pool that every other dependency uses.
Cascades are self-sustaining. The original trigger can resolve minutes before the cascade stops. Recovery requires draining queues, resetting circuit breakers, and letting thread pools clear. This is why "just fix the database" doesn't immediately fix everything.

The Mysterious Latency Spike — when one service is slow
Out of Memory — when thread pool exhaustion leads to OOM
Connection Refused — what happens when the cascade reaches your clients