🎓 Mission Debrief: Stateful App Data Loss

What Happened

You were using a Deployment for a database, which is designed for stateless applications.

Databases are stateful workloads that need: - Stable, predictable pod names - Persistent storage that follows the pod - Ordered startup and shutdown - Stable network identities

Deployments don't provide any of these guarantees, which can lead to data loss, corruption, and split-brain scenarios in databases!

How Kubernetes Behaved

Deployment behavior (wrong for databases):

Deployment creates pods with random names:
- database-7d8f9c-abc12
- database-7d8f9c-xyz78
- database-7d8f9c-mno34

Pod deleted → New pod created with DIFFERENT name:
- database-7d8f9c-pqr56  # Different name!

Result:
❌ No way to identify "primary" vs "replica"
❌ Configuration tied to pod name doesn't work
❌ Persistent volume can't follow pod identity
❌ Can't do ordered operations (start primary first)

StatefulSet behavior (correct for databases):

StatefulSet creates pods with stable ordinal names:
- database-0  (primary)
- database-1  (replica)
- database-2  (replica)

Pod deleted → New pod created with SAME name:
- database-0  # Same name! Can reattach storage

Result:
✅ database-0 is always the primary
✅ Persistent volume for database-0 stays with it
✅ Ordered startup: 0 → 1 → 2
✅ Stable network ID: database-0.database-service

The Correct Mental Model

Stateless vs Stateful Workloads

Stateless (use Deployment):

Web Server Example:
┌─────────┐  ┌─────────┐  ┌─────────┐
│ web-abc │  │ web-xyz │  │ web-mno │
└─────────┘  └─────────┘  └─────────┘
     ↓            ↓            ↓
  [No local state - all identical]

Any pod can handle any request
Pods are interchangeable
No need for stable identity
Can scale up/down instantly

Stateful (use StatefulSet):

Database Example:
┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│ database-0   │  │ database-1   │  │ database-2   │
│ (PRIMARY)    │  │ (REPLICA)    │  │ (REPLICA)    │
└──────┬───────┘  └──────┬───────┘  └──────┬───────┘
       │                  │                  │
       ↓                  ↓                  ↓
    [PV-0]            [PV-1]            [PV-2]

database-0 is ALWAYS the primary
Each pod has its own persistent storage
Pods are NOT interchangeable
Ordered operations required

StatefulSet Guarantees

1. Stable Pod Names:

Deployment:     myapp-7d8f9c-abc12  (random hash)
StatefulSet:    myapp-0             (stable ordinal)

2. Stable Network Identity:

DNS for StatefulSet pods:
- database-0.database-service.k8squest.svc.cluster.local
- database-1.database-service.k8squest.svc.cluster.local
- database-2.database-service.k8squest.svc.cluster.local

Even if pod is deleted and recreated, DNS name stays the same!

3. Ordered Startup:

StatefulSet starts pods in order:
1. Create database-0, wait until Ready
2. Create database-1, wait until Ready
3. Create database-2, wait until Ready

This is critical for databases (primary must start first)!

4. Ordered Shutdown:

StatefulSet deletes pods in reverse order:
1. Delete database-2, wait until Terminated
2. Delete database-1, wait until Terminated
3. Delete database-0, wait until Terminated

This prevents data loss (replicas stop before primary)!

5. Persistent Volume Per Pod:

# StatefulSet with volumeClaimTemplates
apiVersion: apps/v1
kind: StatefulSet
spec:
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

Creates:
- PVC: data-database-0 → Pod: database-0
- PVC: data-database-1 → Pod: database-1
- PVC: data-database-2 → Pod: database-2

If database-0 pod is deleted, new database-0 pod gets same PVC!

When to Use Each

Use Deployment: - Web servers - API backends (stateless) - Batch jobs (stateless) - Microservices (no local state) - Cache layers (can lose data)

Use StatefulSet: - Databases (PostgreSQL, MySQL, MongoDB) - Message queues (Kafka, RabbitMQ) - Distributed caches (Redis cluster) - Consensus systems (etcd, ZooKeeper) - Any app that needs stable identity

Real-World Incident Example

Company: Financial services SaaS (10K business customers) Impact: 3-hour database outage, data corruption, $5M+ in damages Cost: $5M in SLA penalties + emergency restoration costs + customer churn

What happened:

A developer was setting up a PostgreSQL database in Kubernetes. They found an example Deployment configuration and used it, not realizing it was for a simple demo (not production).

Broken configuration:

apiVersion: apps/v1
kind: Deployment  # ❌ WRONG for database!
metadata:
  name: postgres
spec:
  replicas: 3  # Primary + 2 replicas
  template:
    spec:
      containers:
      - name: postgres
        image: postgres:14
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: data
        emptyDir: {}  # ❌ LOSES DATA ON POD RESTART!

The cascade (Friday, 3 PM - month-end closing):

15:00 - Kubernetes node experiences memory pressure
15:00 - Kubelet evicts postgres pod (postgres-abc123) from node
15:00 - Deployment creates new pod (postgres-xyz789) on different node
15:00 - New pod starts with EMPTY emptyDir volume
15:00 - PostgreSQL initializes NEW empty database ❌
15:01 - Application connects to new pod
15:01 - Application thinks database is empty
15:01 - Application starts fresh (creates schema)
15:02 - OLD pod (postgres-abc123) still running on original node
15:02 - TWO postgres instances, DIFFERENT data ❌
15:05 - Service load balances between both pods
15:05 - Half of requests see old data, half see new (empty) data
15:05 - Application in complete chaos
15:05 - Users report: "My data disappeared!"
15:10 - Support tickets flooding in (500 tickets in 5 minutes)
15:15 - Emergency escalation
15:20 - Team identifies two postgres pods with different data
15:25 - Realize: Using Deployment with emptyDir = disaster
15:30 - Immediate decision: Shut down all postgres pods
15:30 - Application completely down ❌
15:45 - Start recovery process from backups
16:00 - Determine which data is canonical (complex!)
16:30 - Restore from backup (1 hour old)
17:00 - Manually merge data from two conflicting pods
18:00 - Database back online
18:00 - But 1 hour of transactions lost + data inconsistencies

Why it was catastrophic: - Month-end closing - Businesses processing critical financial transactions - Data loss - emptyDir doesn't persist across pod restarts - Split-brain - Two postgres instances with different data - No recovery plan - Unprepared for this scenario - Customer impact - Lost invoices, incomplete transactions - Compliance issues - Financial data integrity compromised

What users experienced:

15:00 - Working normally
15:05 - "Where did my invoice go?" (connected to new pod with empty DB)
15:06 - "Why is my data from yesterday?" (connected to old pod)
15:10 - "Application is completely broken!"
15:30 - "Service Unavailable" (all pods shut down for recovery)
18:00 - "We've lost the last hour of work" (restore from 1hr-old backup)

Root cause: 1. Using Deployment instead of StatefulSet - No stable identity 2. emptyDir instead of PersistentVolume - Data not persisted 3. No testing of failure scenarios - Never tested pod eviction 4. No monitoring of database state - Didn't detect split-brain 5. No backup validation - Backup was 1 hour old (too infrequent)

The correct configuration:

apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  clusterIP: None  # Headless service
  selector:
    app: postgres
---
apiVersion: apps/v1
kind: StatefulSet  # ✅ CORRECT for database
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 1  # Start with single instance (avoid replication complexity)
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:14
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
          subPath: postgres  # Important for postgres
  volumeClaimTemplates:  # ✅ Persistent storage
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: ssd  # Use appropriate storage class
      resources:
        requests:
          storage: 100Gi

Process improvements: 1. Architecture review for all stateful workloads 2. Mandatory StatefulSet for databases policy 3. Chaos testing - Regular pod eviction tests 4. Monitoring - Alert on pod name changes (indicates Deployment, not StatefulSet) 5. Backup frequency - Every 15 minutes instead of hourly

Lessons learned: 1. Never use Deployment for databases - Always use StatefulSet 2. Never use emptyDir for stateful data - Always use PersistentVolume 3. Test failure scenarios - Pod eviction, node failure, etc. 4. Monitor database identity - Alert if postgres pods have random names 5. Have rollback plan - Know how to recover from data corruption

Commands You Mastered

# Create StatefulSet
kubectl apply -f statefulset.yaml

# Check StatefulSet status
kubectl get statefulset -n <namespace>

# Get pod names (should be stable: app-0, app-1, app-2)
kubectl get pods -n <namespace> -l app=<label>

# Describe StatefulSet
kubectl describe statefulset <name> -n <namespace>

# Check pod DNS names
kubectl run -it --rm debug --image=busybox --restart=Never -n <namespace> -- nslookup database-0.database-service

# Scale StatefulSet (scales in order)
kubectl scale statefulset <name> --replicas=5 -n <namespace>

# Delete StatefulSet (keeps PVCs by default)
kubectl delete statefulset <name> -n <namespace>

# Delete StatefulSet AND PVCs
kubectl delete statefulset <name> -n <namespace> --cascade=orphan
kubectl delete pvc --all -n <namespace>

# Check PVCs created by StatefulSet
kubectl get pvc -n <namespace>

# Restart pod (StatefulSet recreates with same name)
kubectl delete pod <name>-0 -n <namespace>
# Watch it recreate with same name!

# Check pod startup order
kubectl get pods -n <namespace> -l app=<label> -w

Best Practices for StatefulSets

✅ DO:

1. Use StatefulSet for stateful workloads:

   kind: StatefulSet  # For databases, message queues
   

2. Always use PersistentVolumes:

   volumeClaimTemplates:
   - metadata:
       name: data
     spec:
       resources:
         requests:
           storage: 10Gi
   

3. Use headless Service:

   apiVersion: v1
   kind: Service
   spec:
     clusterIP: None  # Required for StatefulSet DNS
   

4. Set appropriate update strategy:

   updateStrategy:
     type: RollingUpdate  # Or OnDelete for manual control
     rollingUpdate:
       partition: 0  # Update all pods
   

5. Use readiness probes:

   readinessProbe:
     exec:
       command: ["pg_isready", "-U", "postgres"]
   

❌ DON'T:

1. Don't use Deployment for databases:

   # ❌ WRONG
   kind: Deployment
   # ✅ CORRECT
   kind: StatefulSet
   

2. Don't use emptyDir for stateful data:

   # ❌ WRONG - loses data on pod restart
   volumes:
   - name: data
     emptyDir: {}
   
   # ✅ CORRECT - persistent storage
   volumeClaimTemplates:
   - metadata:
       name: data
   

3. Don't forget serviceName:

   # Required for StatefulSet!
   spec:
     serviceName: "my-service"
   

4. Don't manually change pod ordinals:

   # Bad: Rename database-1 to database-0
   # Breaks StatefulSet ordering!
   

StatefulSet Advanced Patterns

Pattern 1: Database Primary/Replica

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  replicas: 3
  template:
    spec:
      initContainers:
      - name: init-mysql
        image: mysql:5.7
        command:
        - bash
        - "-c"
        - |
          set -ex
          # Primary is pod-0, replicas are pod-1, pod-2, etc.
          [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
          ordinal=${BASH_REMATCH[1]}
          if [[ $ordinal -eq 0 ]]; then
            cp /mnt/config-map/primary.cnf /mnt/conf.d/
          else
            cp /mnt/config-map/replica.cnf /mnt/conf.d/
          fi

Pattern 2: Ordered Dependency

# Start pods in order, each depends on previous
apiVersion: apps/v1
kind: StatefulSet
spec:
  podManagementPolicy: OrderedReady  # Default
  # vs Parallel (all pods start at once)

Pattern 3: Canary Updates

# Update only some pods (testing)
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2  # Only update pods >= ordinal 2
                     # Pods 0 and 1 stay on old version

What's Next?

You've learned the critical difference between Deployments (stateless) and StatefulSets (stateful), and why databases need stable identities.

Next level (FINAL LEVEL OF WORLD 2!): ReplicaSet chaos! You'll learn why manually creating ReplicaSets without a Deployment causes management nightmares.

Key takeaway: Never use Deployments for stateful workloads like databases. Always use StatefulSets with PersistentVolumes for data persistence and stable pod identities!