🎓 Mission Debrief: Traffic to Unready Pods

What Happened

Your pods were receiving traffic before they were ready to handle it, causing 502 Bad Gateway errors for users.

The root cause: No readiness probe configured.

Without a readiness probe, Kubernetes assumes that as soon as a pod's status is "Running", it's ready to receive traffic. But "Running" just means the container process started—it doesn't mean the application is initialized and ready!

Your app had a 20-second startup delay (simulated with sleep 20 in postStart hook). During those 20 seconds, pods were added to the Service endpoints and received real user traffic, but they couldn't handle it yet.

How Kubernetes Behaved

Without readiness probe (BROKEN):

Pod lifecycle:
1. Pod created
2. Container starts → Status: Running
3. ✅ Immediately added to Service endpoints (bad!)
4. Receives traffic from Service
5. App still starting up (sleep 20 running)
6. Returns 502 Bad Gateway to users ❌
7. After 20s, app is actually ready
8. Too late - users already got errors!

With readiness probe (FIXED):

Pod lifecycle:
1. Pod created
2. Container starts → Status: Running
3. ⏳ NOT added to Service yet (readiness probe not passed)
4. Wait initialDelaySeconds (22s)
5. Check readiness probe (HTTP GET /:8080)
6. Get 200 OK response ✅
7. Mark pod as Ready
8. ✅ NOW added to Service endpoints
9. Receives traffic, app is ready, users happy!

The Correct Mental Model

Liveness vs Readiness: The Critical Difference

Aspect	Liveness Probe	Readiness Probe
Question	"Is the container alive?"	"Is the container ready for traffic?"
Action on failure	Restart the container	Remove from Service endpoints
Use case	Detect deadlocks, infinite loops	Prevent traffic during startup/overload
Failure is	Fatal (needs restart)	Temporary (will recover)
Example	Process crashed	Database connection not ready

When to Use Each Probe

Use Liveness Probe when: - Detecting application deadlocks - Process is stuck in infinite loop - Memory leak has frozen the app - Recovery method: Restart

Use Readiness Probe when: - Application needs time to load data into cache - Waiting for database connection - Loading configuration files - Temporary overload (too many requests) - Recovery method: Wait, don't send traffic

Readiness Probe Configuration

readinessProbe:
  # HTTP probe (most common)
  httpGet:
    path: /ready        # Endpoint that checks if app is ready
    port: 8080
    httpHeaders:
    - name: X-Custom-Header
      value: health-check

  initialDelaySeconds: 10   # How long to wait before first check
  periodSeconds: 5          # How often to check
  timeoutSeconds: 3         # Max time to wait for response
  successThreshold: 1       # How many successes needed (usually 1)
  failureThreshold: 3       # How many failures before marking unready

Alternative probe types:

# TCP socket probe (just check if port is listening)
readinessProbe:
  tcpSocket:
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

# Exec probe (run a command)
readinessProbe:
  exec:
    command:
    - cat
    - /tmp/ready
  initialDelaySeconds: 5
  periodSeconds: 5

Real-World Incident Example

Company: E-commerce platform (Black Friday sale) Impact: 15 minutes of 30% error rate, 500K failed requests Cost: $2.8M in lost sales + reputation damage

What happened:

The team deployed a new version of their product service with caching enabled. The cache took 45 seconds to warm up (loading 10GB of product data from database).

Deployment configuration:

# BROKEN configuration
spec:
  replicas: 50
  template:
    spec:
      containers:
      - name: product-service
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 10
        # ❌ NO readiness probe!

The cascade (Black Friday, 2:00 PM peak traffic):

14:00:00 - Deployment starts (RollingUpdate, maxSurge: 25%)
14:00:05 - First 12 new pods start (Running status)
14:00:05 - Pods immediately added to Service endpoints ❌
14:00:06 - Load balancer sends traffic to new pods
14:00:06 - Cache not ready → queries hit database directly
14:00:07 - Database overloaded (50x normal load)
14:00:10 - Database connection pool exhausted
14:00:11 - New pods return 503 Service Unavailable
14:00:12 - Old pods also struggle (shared database)
14:00:15 - 30% of all requests failing (50K req/s × 30% = 15K errors/s)
14:00:15 - Alerts fire, team paged
14:03:00 - Team identifies issue
14:05:00 - Emergency rollback initiated
14:08:00 - Old version fully restored
14:12:00 - Database recovers
14:15:00 - Service fully operational

Why it was catastrophic: 1. Black Friday traffic - 10x normal load 2. No readiness probe - Unready pods got traffic immediately 3. Cache warmup hits database - Overloaded shared resource 4. Cascading failure - Database overload affected old pods too

The fix:

readinessProbe:
  httpGet:
    path: /ready  # New endpoint that checks cache status
    port: 8080
  initialDelaySeconds: 50  # Wait for 45s cache warmup + 5s buffer
  periodSeconds: 10
  failureThreshold: 3

New /ready endpoint in code:

func readyHandler(w http.ResponseWriter, r *http.Request) {
    if !cache.IsWarmedUp() {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

Lessons learned: 1. Always use readiness probes for apps with startup time 2. Test under production-like load before Black Friday 3. Implement health endpoints that check actual readiness (not just "is process running") 4. Use progressive rollouts to catch issues early 5. Monitor error rates during deployments

Commands You Mastered

# Check pod readiness status
kubectl get pods -n <namespace>
# Look at READY column: "0/1" = not ready, "1/1" = ready

# Check which pods are receiving traffic
kubectl get endpoints <service-name> -n <namespace>
# Shows IP addresses of READY pods

# Describe pod to see readiness probe status
kubectl describe pod <name> -n <namespace>
# Look for "Readiness" in Conditions section

# Check deployment readiness configuration
kubectl get deployment <name> -n <namespace> -o yaml | grep -A 10 readinessProbe

# Watch pods become ready in real-time
kubectl get pods -n <namespace> -l app=<label> -w

# Edit deployment to add readiness probe
kubectl edit deployment <name> -n <namespace>

# Check deployment rollout status
kubectl rollout status deployment/<name> -n <namespace>

Best Practices for Readiness Probes

✅ DO:

1. Always configure readiness probes:

   # Every deployment should have this!
   readinessProbe:
     httpGet:
       path: /ready
       port: 8080
   

2. Make readiness checks meaningful:

   // Good readiness check
   func ready() bool {
       return database.IsConnected() &&
              cache.IsWarmed() &&
              configLoaded
   }
   
   // Bad readiness check
   func ready() bool {
       return true  // Useless!
   }
   

3. Use initialDelaySeconds appropriately:

   initialDelaySeconds: 30  # Give app time to start
   

4. Set reasonable failure thresholds:

   failureThreshold: 3  # Don't remove from service immediately
   

5. Use different endpoints for liveness and readiness:

   livenessProbe:
     httpGet:
       path: /healthz  # Is process alive?
   readinessProbe:
     httpGet:
       path: /ready    # Is app ready for traffic?
   

❌ DON'T:

1. Don't skip readiness probes:

   # Bad: Only liveness, no readiness
   livenessProbe: ...
   # Good: Both probes
   livenessProbe: ...
   readinessProbe: ...
   

2. Don't use same probe for both:

   # Bad: Same endpoint for both
   livenessProbe:
     httpGet:
       path: /health
   readinessProbe:
     httpGet:
       path: /health  # Too simple!
   
   # Good: Different logic
   livenessProbe:
     httpGet:
       path: /alive     # Just checks process
   readinessProbe:
     httpGet:
       path: /ready     # Checks dependencies
   

3. Don't make readiness checks too strict:

   // Bad: Unready if any dependency has slight issue
   func ready() bool {
       return db.Ping() == nil &&
              redis.Ping() == nil &&
              elasticsearch.Ping() == nil &&
              kafka.IsHealthy() &&
              apiGateway.IsReachable()
       // One hiccup = pod removed from service!
   }
   
   // Good: Only check critical dependencies
   func ready() bool {
       return db.IsConnected()  // Only critical dependency
   }
   

4. Don't forget the readiness endpoint:

   readinessProbe:
     httpGet:
       path: /ready  # This endpoint must exist in your app!
   

Readiness Probe Patterns

Pattern 1: Database Dependency

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 3

// In your app
func readyHandler(w http.ResponseWriter, r *http.Request) {
    if err := db.Ping(); err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        log.Printf("Not ready: database not connected: %v", err)
        return
    }
    w.WriteHeader(http.StatusOK)
}

Pattern 2: Cache Warmup

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 60  # Cache needs 1 minute to warm
  periodSeconds: 10

# In your app
@app.route('/ready')
def ready():
    if not cache.is_warmed_up():
        return '', 503  # Not ready yet
    return '', 200  # Ready!

Pattern 3: Configuration Loading

readinessProbe:
  exec:
    command:
    - cat
    - /tmp/config-loaded
  initialDelaySeconds: 5
  periodSeconds: 5

# In your startup script
load_config() {
    # Load configuration
    ...
    # Signal readiness
    touch /tmp/config-loaded
}

Debugging Readiness Issues

Step-by-step troubleshooting:

# 1. Check if pods are ready
kubectl get pods -n <namespace>

# 2. If READY is "0/1", check why
kubectl describe pod <name> -n <namespace>
# Look at Conditions → Ready = False, Reason = ...

# 3. Check if readiness probe is failing
kubectl describe pod <name> -n <namespace> | grep -A 20 "Readiness"

# 4. Test the readiness endpoint manually
kubectl port-forward pod/<name> 8080:8080 -n <namespace>
curl http://localhost:8080/ready  # Should return 200

# 5. Check the logs
kubectl logs <pod-name> -n <namespace>

# 6. If no readiness probe configured, add one
kubectl edit deployment <name> -n <namespace>

What's Next?

You've learned the critical difference between liveness and readiness probes, and how to prevent traffic from hitting unready pods.

Next level: HorizontalPodAutoscaler! You'll learn how to automatically scale pods based on CPU/memory metrics.

Key takeaway: Readiness probes protect your users from unready pods. Always configure them for production deployments!