🎓 Mission Debrief: Zero-Downtime Deployment Failure

What Happened

Your deployment was configured with maxUnavailable: 100% and maxSurge: 0, which allowed Kubernetes to terminate all pods simultaneously during a rolling update.

This caused complete service outage every time you deployed a new version!

This is a critical production mistake that can turn routine deployments into incidents.

How Kubernetes Behaved

With broken config (maxUnavailable: 100%, maxSurge: 0):

Rolling update starts (3 replicas → new version)
    ↓
maxUnavailable: 100% means can terminate 100% of 3 = all 3 pods
maxSurge: 0 means cannot create extra pods
    ↓
Step 1: Terminate all 3 old pods ❌
    ↓
Step 2: Create 3 new pods
    ↓
Step 3: Wait for new pods to be ready
    ↓
Service has 0 pods for 10-30 seconds → DOWNTIME!

With fixed config (maxUnavailable: 1, maxSurge: 1):

Rolling update starts (3 replicas → new version)
    ↓
maxUnavailable: 1 means max 1 pod down (keep 2 running)
maxSurge: 1 means can create 1 extra pod (total 4)
    ↓
Step 1: Create 1 new pod (now 4 total)
Step 2: Wait for new pod to be ready
Step 3: Terminate 1 old pod (back to 3 total, 2 old + 1 new)
Step 4: Create another new pod (4 total again)
Step 5: Wait for it to be ready
Step 6: Terminate another old pod
Step 7-8: Repeat for last pod
    ↓
Service ALWAYS has at least 2 pods running → ZERO DOWNTIME! ✅

The Correct Mental Model

RollingUpdate Parameters Explained

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 1  # or "25%" - Max pods that can be down
    maxSurge: 1        # or "25%" - Max extra pods during rollout

maxUnavailable: Maximum number of pods that can be unavailable during the update.

With 3 replicas:
- maxUnavailable: 0   → Must keep all 3 running (requires maxSurge > 0)
- maxUnavailable: 1   → Can take down 1, keep 2 running (safe)
- maxUnavailable: 2   → Can take down 2, keep only 1 running (risky)
- maxUnavailable: 3   → Can take down all 3 (DOWNTIME!)
- maxUnavailable: 100% → Same as 3 (DOWNTIME!)

maxSurge: Maximum number of pods that can be created above desired replicas.

With 3 replicas:
- maxSurge: 0   → Cannot create extras (must terminate first)
- maxSurge: 1   → Can have 4 pods during rollout
- maxSurge: 2   → Can have 5 pods during rollout
- maxSurge: 100% → Can have 6 pods during rollout (2x)

Common Configuration Patterns

Conservative (safest, slower):

maxUnavailable: 0     # Never take down any pod
maxSurge: 1           # Create one new pod at a time
# Guarantees: 100% availability, slowest rollout

Balanced (recommended for most apps):

maxUnavailable: 1     # Take down 1 pod at a time
maxSurge: 1           # Allow 1 extra pod
# Guarantees: High availability, reasonable speed

Aggressive (faster, less safe):

maxUnavailable: 25%   # Can take down 25% of pods
maxSurge: 25%         # Can create 25% extra pods
# Guarantees: Faster rollout, some availability risk

DANGEROUS (causes downtime):

maxUnavailable: 100%  # Can take down ALL pods ❌
maxSurge: 0          # Cannot create extras ❌
# Result: Complete outage during rollout!

Using Percentages

# For 10 replicas:
maxUnavailable: 25%  # = 2.5 → rounds down to 2 pods
maxSurge: 25%        # = 2.5 → rounds up to 3 pods

# For 3 replicas:
maxUnavailable: 33%  # = 0.99 → rounds down to 0 pods
maxSurge: 33%        # = 0.99 → rounds up to 1 pod

Rounding rules: - maxUnavailable: rounds down (conservative, keeps more pods) - maxSurge: rounds up (generous, allows more extra pods)

Real-World Incident Example

Company: SaaS CRM platform (100K business users) Impact: 8-minute complete outage during business hours Cost: $450K in SLA credits + major customer churn

What happened:

A DevOps engineer was trying to optimize resource usage. They noticed that during deployments, Kubernetes temporarily created extra pods (due to maxSurge: 1), using extra memory.

To "save resources," they changed the deployment strategy:

# Before (safe)
strategy:
  rollingUpdate:
    maxUnavailable: 1
    maxSurge: 1

# After (BROKEN)
strategy:
  rollingUpdate:
    maxUnavailable: 100%  # "No downtime if we update fast!"
    maxSurge: 0           # "No extra resource usage!"

The engineer's logic: "If we update really fast, there won't be downtime!"

The cascade (Tuesday, 2 PM):

14:00 - Deploy new feature (routine update)
14:00 - RollingUpdate starts for 'auth-service' (5 replicas)
14:00:05 - Kubernetes terminates ALL 5 pods (maxUnavailable: 100%)
14:00:06 - Starts creating 5 new pods
14:00:06 - Service has 0 healthy endpoints
14:00:06 - All users see "Service Unavailable"
14:00:08 - First new pod ready (2 seconds startup)
14:00:10 - Second pod ready
14:00:12 - Third pod ready
14:00:13 - Fourth pod ready
14:00:14 - All 5 pods ready
14:08 - Service fully restored

Why it was catastrophic: - Authentication service - When it's down, entire platform is unusable - Business hours - 2 PM = peak usage time - 100K users affected - All simultaneously logged out - No warning - Routine deployment caused incident - 8 minutes downtime - For what should be zero-downtime deployment

Customer impact:

14:00 - Users in middle of important sales calls
14:00 - CRM goes down, all data inaccessible
14:01 - Support tickets flood in (1,200 tickets in 1 minute)
14:02 - Social media complaints start
14:05 - Major customers (Fortune 500) threaten to switch
14:08 - Service restored
14:15 - Damage control begins
14:30 - CEO personally calls top 10 customers to apologize

The fix:

1. Immediate rollback of deployment strategy:

   strategy:
     rollingUpdate:
       maxUnavailable: 1  # Restore safe config
       maxSurge: 1
   

2. Deployment validation added to CI/CD:

   #!/bin/bash
   # Check for dangerous rollout strategies
   MAX_UNAVAIL=$(yq '.spec.strategy.rollingUpdate.maxUnavailable' deployment.yaml)
   if [[ "$MAX_UNAVAIL" == "100%" ]] || [[ "$MAX_UNAVAIL" -ge $(yq '.spec.replicas' deployment.yaml) ]]; then
       echo "ERROR: Dangerous maxUnavailable detected!"
       exit 1
   fi
   

3. Policy enforcement with OPA (Open Policy Agent):

   deny[msg] {
       input.kind == "Deployment"
       maxUnavail := input.spec.strategy.rollingUpdate.maxUnavailable
       maxUnavail >= input.spec.replicas
       msg := "maxUnavailable must be less than replicas to prevent downtime"
   }
   

Lessons learned: 1. Never sacrifice availability for resource savings - Extra pods during 30-second rollout are worth it 2. Test rollout strategies in staging - With actual traffic 3. Validate deployment configs - Automated checks in CI/CD 4. Monitor during deployments - Alert if healthy pods drop to zero 5. Know your critical services - Auth, API gateway need extra safety

Commands You Mastered

# Check deployment strategy
kubectl get deployment <name> -n <namespace> -o yaml | grep -A 5 strategy

# Edit deployment strategy
kubectl edit deployment <name> -n <namespace>

# Watch rollout in progress
kubectl rollout status deployment/<name> -n <namespace>

# Watch pods during rollout
kubectl get pods -n <namespace> -l app=<label> -w

# Trigger a rollout (for testing)
kubectl set image deployment/<name> <container-name>=new-image:tag -n <namespace>

# Pause a problematic rollout
kubectl rollout pause deployment/<name> -n <namespace>

# Resume after fixing
kubectl rollout resume deployment/<name> -n <namespace>

# Rollback if deployment went bad
kubectl rollout undo deployment/<name> -n <namespace>

# Check rollout history
kubectl rollout history deployment/<name> -n <namespace>

Best Practices for Rollout Strategies

✅ DO:

1. Use safe defaults:

   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxUnavailable: 1
       maxSurge: 1
   

2. For critical services, be extra conservative:

   # Auth, payment, core API services
   strategy:
     rollingUpdate:
       maxUnavailable: 0  # Never take down any pod
       maxSurge: 1        # Create new before removing old
   

3. Use percentages for large deployments:

   # For 100+ replicas
   strategy:
     rollingUpdate:
       maxUnavailable: 10%  # Update 10 pods at a time
       maxSurge: 10%
   

4. Add readiness probes (essential for RollingUpdate):

   readinessProbe:
     httpGet:
       path: /ready
       port: 8080
   # Without this, new pods get traffic before ready!
   

5. Monitor deployments:

   # Alert if available pods drops too low during rollout
   - alert: DeploymentBelowMinimumDuringRollout
     expr: kube_deployment_status_replicas_available < (kube_deployment_spec_replicas * 0.8)
   

❌ DON'T:

1. Never use maxUnavailable: 100%:

   # NEVER do this!
   maxUnavailable: 100%
   

2. Don't set both to 0:

   # Invalid configuration!
   maxUnavailable: 0
   maxSurge: 0
   # At least one must be > 0
   

3. Don't skip readiness probes:

   # Without readiness probe:
   # New pods get traffic immediately, might fail!
   

4. Don't use aggressive settings for critical services:

   # Too risky for auth/payment services
   maxUnavailable: 50%
   

Rollout Strategy Decision Tree

Is this a critical service (auth, payment, core API)?
├─ YES: Use maxUnavailable: 0, maxSurge: 1
│       (Create new before removing old)
└─ NO: Continue ↓

Do you have many replicas (>20)?
├─ YES: Use percentages
│       maxUnavailable: 10%
│       maxSurge: 10%
└─ NO: Continue ↓

Do you have enough resources for extra pods?
├─ YES: Use maxUnavailable: 1, maxSurge: 1
│       (Balanced, recommended)
└─ NO: Use maxUnavailable: 1, maxSurge: 0
       (Resource-constrained, slower rollout)

What's Next?

You've learned how to configure rollout strategies for zero-downtime deployments.

Next level: PodDisruptionBudgets! You'll learn how to protect your pods from being evicted during cluster maintenance.

Key takeaway: Always ensure maxUnavailable is less than your total replicas, and prefer having maxSurge > 0 for smoother rollouts!