🎓 LEVEL 33 DEBRIEF: PV/PVC Access Modes¶

Congratulations! You've successfully fixed the access mode configuration! Understanding access modes is critical for shared storage scenarios.

📊 What You Fixed¶

The Problem:

# PersistentVolume & PersistentVolumeClaim
accessModes:
  - ReadWriteOnce  # ❌ Only one node can mount at a time!

The Solution:

# PersistentVolume & PersistentVolumeClaim
accessModes:
  - ReadWriteMany  # ✅ Multiple nodes can mount simultaneously

⚠️ About This Level¶

Important Context: This level runs on Kind (single-node cluster), so even with ReadWriteOnce, all 3 pods successfully start because they're all on the same node.

In a real multi-node production cluster: - ReadWriteOnce with 3 replicas would cause pods on different nodes to fail mounting - Only pods on the first node would start; others would be Pending with volume attach errors

This level teaches TWO important concepts: 1. Access Mode selection - choosing RWO vs RWX for your workload 2. PVC immutability - you'll discover that changing accessModes requires deleting and recreating the PVC!

🔍 Understanding Access Modes¶

The Three Access Modes¶

Mode	Short	Description	Use Case
ReadWriteOnce	RWO	One node, read-write	Single pod or pods on same node
ReadOnlyMany	ROX	Many nodes, read-only	Static content distribution
ReadWriteMany	RWX	Many nodes, read-write	Shared storage across pods

Critical: Node vs Pod¶

ReadWriteOnce = One NODE, not one pod!

❌ Common Misconception:
"ReadWriteOnce means only one pod can use it"

✅ Reality:
"ReadWriteOnce allows read-write access from a single node at a time,
 but multiple pods on that node can share it"

Example Scenarios:

# Scenario 1: Multiple pods on SAME node (ReadWriteOnce works)
Node1: [Pod-A] [Pod-B] [Pod-C]  # ✅ All can mount RWO volume
Node2: [Pod-D]                   # ❌ Cannot mount same RWO volume

# Scenario 2: Pods on DIFFERENT nodes (Need ReadWriteMany)
Node1: [Pod-A]  # ✅ Can mount RWX volume
Node2: [Pod-B]  # ✅ Can mount same RWX volume
Node3: [Pod-C]  # ✅ Can mount same RWX volume

🎯 When to Use Each Mode¶

Use ReadWriteOnce (RWO) When:¶

Single-instance applications

# Databases, single-replica apps
replicas: 1
accessModes:
  - ReadWriteOnce

StatefulSets (each pod gets own PVC)

# Each pod has dedicated storage
volumeClaimTemplates:
- metadata:
    name: data
  spec:
    accessModes: [ReadWriteOnce]

Node affinity guaranteed

# All pods scheduled to same node
affinity:
  podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector: ...
      topologyKey: kubernetes.io/hostname

Use ReadWriteMany (RWX) When:¶

Shared web content

# Multiple web servers serving same content
kind: Deployment
replicas: 5
volumeMounts:
- name: web-content
  mountPath: /usr/share/nginx/html

Distributed processing

# Multiple workers reading same data
kind: Job
parallelism: 10
volumeMounts:
- name: input-data
  mountPath: /data
  readOnly: true

Shared configuration

# Multiple apps reading shared config
kind: Deployment
replicas: 3
volumeMounts:
- name: shared-config
  mountPath: /etc/config

Use ReadOnlyMany (ROX) When:¶

Static assets distribution

# Distribute images, CSS, JS files
accessModes:
  - ReadOnlyMany
# Prevents accidental modifications

Shared reference data

# ML models, lookup tables
volumeMounts:
- name: ml-models
  mountPath: /models
  readOnly: true

🗄️ Storage Provider Support¶

Not all storage types support all access modes!

⚠️ Important Note About This Level¶

In this simulation: - We use hostPath with ReadWriteMany for learning purposes - This works in Kind (single-node cluster) where all pods land on the same node - The hostPath represents shared network storage (like NFS/EFS)

In production: - hostPath is node-local and does NOT support true ReadWriteMany - For real RWX, you need network-attached storage (NFS, EFS, CephFS, etc.) - This level teaches the concept of access modes in a multi-pod scenario

Why This Distinction Matters:

# ❌ In multi-node production cluster with hostPath:
accessModes: [ReadWriteMany]  # Kubernetes accepts this...
hostPath:                      # But hostPath is node-local!
  path: /data
# Result: Pods on different nodes can't access the same data

# ✅ In production for true RWX:
accessModes: [ReadWriteMany]
nfs:                          # Use network storage
  server: nfs.example.com
  path: /shared
# Result: All pods can truly share the data

Cloud Provider Storage:¶

AWS EBS - ✅ ReadWriteOnce - ❌ ReadWriteMany - ✅ ReadOnlyMany

AWS EFS - ✅ ReadWriteOnce - ✅ ReadWriteMany - ✅ ReadOnlyMany

Google Persistent Disk - ✅ ReadWriteOnce - ❌ ReadWriteMany (except for ROX) - ✅ ReadOnlyMany

Google Filestore - ✅ ReadWriteOnce - ✅ ReadWriteMany - ✅ ReadOnlyMany

Azure Disk - ✅ ReadWriteOnce - ❌ ReadWriteMany - ✅ ReadOnlyMany

Azure Files - ✅ ReadWriteOnce - ✅ ReadWriteMany - ✅ ReadOnlyMany

On-Premise Storage:¶

NFS - ✅ ReadWriteOnce - ✅ ReadWriteMany - ✅ ReadOnlyMany

hostPath (LOCAL NODE ONLY) - ✅ ReadWriteOnce (single node) - ❌ ReadWriteMany (NOT network storage!) - ✅ ReadOnlyMany (single node)

iSCSI - ✅ ReadWriteOnce - ❌ ReadWriteMany - ✅ ReadOnlyMany

Ceph RBD - ✅ ReadWriteOnce - ❌ ReadWriteMany - ✅ ReadOnlyMany

CephFS - ✅ ReadWriteOnce - ✅ ReadWriteMany - ✅ ReadOnlyMany

GlusterFS - ✅ ReadWriteOnce - ✅ ReadWriteMany - ✅ ReadOnlyMany

Common Access Mode Mistakes¶

Mistake 1: Assuming All Storage Supports RWX¶

# Using cloud block storage
storageClassName: gp2  # AWS EBS
accessModes:
  - ReadWriteMany  # ❌ EBS doesn't support RWX!

# Result: PVC stuck in Pending

Fix: Use appropriate storage class

storageClassName: efs-sc  # AWS EFS
accessModes:
  - ReadWriteMany  # ✅ EFS supports RWX

Mistake 2: PV and PVC Access Mode Mismatch¶

# PersistentVolume
accessModes:
  - ReadWriteOnce

# PersistentVolumeClaim
accessModes:
  - ReadWriteMany  # ❌ Doesn't match PV!

# Result: PVC won't bind to PV

Fix: Match access modes

# Both must have same access mode
accessModes:
  - ReadWriteMany

Mistake 3: Using RWO for Deployment with Multiple Replicas¶

kind: Deployment
replicas: 5  # Pods will likely spread across nodes

volumes:
- persistentVolumeClaim:
    claimName: my-pvc  # accessMode: ReadWriteOnce

# Result: Only pods on first node start, others stuck

Fix: Use ReadWriteMany or reduce to single replica

Mistake 4: Not Checking StorageClass Capabilities¶

# Created custom StorageClass
kind: StorageClass
provisioner: kubernetes.io/aws-ebs  # EBS = block storage

# User tries to use it
accessModes:
  - ReadWriteMany  # ❌ EBS can't do this!

Fix: Verify storage backend capabilities

🔒 Critical Lesson: PVC Spec Immutability¶

You discovered this in the level: Most PVC fields cannot be changed after creation!

What You Can't Change¶

apiVersion: v1
kind: PersistentVolumeClaim
spec:
  accessModes:           # ❌ IMMUTABLE
    - ReadWriteOnce
  storageClassName: gp2  # ❌ IMMUTABLE
  volumeName: pv-123     # ❌ IMMUTABLE

  resources:
    requests:
      storage: 10Gi      # ✅ Can be expanded (if StorageClass allows)

The Error You Saw¶

When you tried to apply the solution without deleting:

$ kubectl apply -f solution.yaml
The PersistentVolumeClaim "shared-pvc" is invalid:
spec: Forbidden: spec is immutable after creation except
resources.requests and volumeAttributesClassName for bound claims

@@ -1,6 +1,6 @@
 {
  "AccessModes": [
-  "ReadWriteOnce"
+  "ReadWriteMany"
  ],

Why This Restriction Exists¶

Storage is already provisioned - PV is created and bound
Data consistency - Changing access mode could cause data corruption
Cloud provider limitations - Underlying storage can't be modified
Volume identity - The bound PV can't change its characteristics

The Fix: Delete and Recreate¶

# Step 1: Delete resources (order matters!)
kubectl delete deployment web-servers -n k8squest
kubectl delete pvc shared-pvc -n k8squest
kubectl delete pv shared-storage

# Step 2: Apply corrected configuration
kubectl apply -f solution.yaml

# Step 3: Verify
kubectl get pvc -n k8squest
kubectl get pods -n k8squest

⚠️ Production Warning¶

In production, deleting a PVC can delete your data!

# Check reclaim policy first!
persistentVolumeReclaimPolicy: Delete   # ⚠️  PV deleted when PVC deleted
persistentVolumeReclaimPolicy: Retain   # ✅ PV kept, data safe

Safe workflow for production: 1. Backup your data first 2. Check PV reclaim policy is Retain 3. Delete PVC (PV remains with data) 4. Create new PVC with correct access mode 5. Manually bind to existing PV (if needed) 6. Restore deployment

What You CAN Change: Volume Expansion¶

# Original PVC
resources:
  requests:
    storage: 10Gi

# Can be increased (if StorageClass allowVolumeExpansion: true)
resources:
  requests:
    storage: 20Gi

# ✅ This works! (kubectl apply)
# PVC automatically expanded

Real-World Impact¶

Why this matters: Many teams learn this the hard way: - Deploy with wrong access mode - Realize mistake after data is written - Can't simply edit and apply - Must plan data migration/backup - Causes downtime and complexity

Best practice: Get access modes right from the start!

🏗️ Real-World Patterns¶

Pattern 1: Separate Storage for Different Needs¶

# StatefulSet with per-pod RWO + shared RWX
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web-app
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        volumeMounts:
        - name: private-data     # Each pod gets own storage
          mountPath: /data
        - name: shared-assets    # All pods share this
          mountPath: /assets
      volumes:
      - name: shared-assets
        persistentVolumeClaim:
          claimName: shared-pvc  # RWX
  volumeClaimTemplates:           # RWO per pod
  - metadata:
      name: private-data
    spec:
      accessModes: [ReadWriteOnce]
      resources:
        requests:
          storage: 10Gi

Pattern 2: Read-Write Leader, Read-Only Followers¶

# Leader pod (read-write)
apiVersion: v1
kind: Pod
metadata:
  name: leader
spec:
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-pvc
      readOnly: false  # Can write

---
# Follower pods (read-only)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: followers
spec:
  replicas: 5
  template:
    spec:
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: data-pvc
          readOnly: true  # Read-only access

Pattern 3: Progressive Migration to Shared Storage¶

# Phase 1: Start with RWO, single replica
kind: Deployment
replicas: 1
volumeClaimTemplate:
  accessModes: [ReadWriteOnce]

# Phase 2: Migrate data to RWX storage
# (manual data migration step)

# Phase 3: Scale with RWX
kind: Deployment
replicas: 10
volumeClaimTemplate:
  accessModes: [ReadWriteMany]

🚨 REAL-WORLD HORROR STORY: The Access Mode Assumption¶

The Incident: $1.2M E-commerce Site Crash¶

Company: Online retail platform Date: Black Friday 2025 Impact: 4 hours downtime during peak sales, $1.2M revenue loss

What Happened¶

Infrastructure team migrated from on-premise to AWS:

# On-premise (worked fine)
storageClassName: nfs-storage
accessModes:
  - ReadWriteMany
# NFS supports RWX ✅

# AWS migration (didn't work)
storageClassName: gp2  # EBS volumes
accessModes:
  - ReadWriteMany  # ❌ EBS doesn't support this!
# PVCs stuck in Pending

The Timeline¶

00:00 - Black Friday migration started (traffic already high) 00:15 - Deployment updated, PVCs provisioned 00:16 - All PVCs stuck in Pending state 00:17 - All web server pods stuck ContainerCreating 00:20 - Site completely down, millions of shoppers affected 01:00 - Emergency rollback started 01:30 - Rollback failed (old PVCs deleted) 02:00 - Quick fix: Change to ReadWriteOnce, reduce to 1 replica 02:30 - Site restored with severely limited capacity 04:00 - EFS deployment complete, full capacity restored

Root Causes¶

Insufficient testing: Migration not tested with production config
Wrong assumption: "All storage is the same in the cloud"
No validation: Didn't check if storage class supported RWX
Poor timing: Major change during peak traffic period
No rollback plan: Couldn't quickly revert to working state

The Fix¶

# Immediate fix
storageClassName: gp2
accessModes:
  - ReadWriteOnce  # EBS supports this
replicas: 1        # Only one pod can run

# Proper fix (deployed 4 hours later)
storageClassName: efs-sc  # AWS EFS
accessModes:
  - ReadWriteMany  # EFS supports this ✅
replicas: 20       # Scale for Black Friday

Lessons Learned¶

Know your storage backend: Different types have different capabilities
Test before production: Especially during critical periods
Validate configurations: Check if storage class supports required access modes
Have rollback plan: Test rollback procedures beforehand
Monitor PVC status: Alert when PVCs stuck in Pending
Document dependencies: Make storage requirements explicit

🛡️ Best Practices¶

1. Document Access Mode Requirements¶

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
  annotations:
    description: "Requires RWX for multi-pod deployment"
    storage-requirements: "Network filesystem (NFS, EFS, or CephFS)"
spec:
  accessModes:
    - ReadWriteMany

2. Validate Storage Class Capabilities¶

# Check what storage classes support
kubectl get storageclass -o custom-columns=\
NAME:.metadata.name,\
PROVISIONER:.provisioner,\
VOLUME-BINDING:.volumeBindingMode

# Test before production
kubectl apply -f test-pvc.yaml
kubectl describe pvc test-pvc
# Check for access mode errors

3. Use Admission Controllers¶

# ValidatingWebhookConfiguration to check compatibility
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: validate-pvc-access-modes
webhooks:
- name: pvc-validator.example.com
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["persistentvolumeclaims"]
  # Validates that requested access mode is supported

4. Monitor PVC Binding¶

# Alert on PVC stuck in Pending
apiVersion: v1
kind: Alert
metadata:
  name: pvc-pending
spec:
  expr: |
    kube_persistentvolumeclaim_status_phase{phase="Pending"} > 0
  for: 5m
  annotations:
    summary: "PVC stuck in Pending state"
    description: "Check if access mode is supported by storage class"

5. Choose Right Storage Class¶

# Define storage classes for different needs
---
# For single-pod applications (cheaper, better performance)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-rwo
  annotations:
    description: "High-performance block storage (RWO only)"
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io2
---
# For multi-pod applications (more expensive, network-based)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: shared-rwx
  annotations:
    description: "Shared network filesystem (RWX supported)"
provisioner: efs.csi.aws.com

🎯 Key Takeaways¶

Access modes are about NODES, not pods - RWO = one node, RWX = many nodes
Not all storage supports all modes - Check your storage backend capabilities
PV and PVC must match - Access modes must be compatible
Choose based on your needs - RWO for single-pod, RWX for shared access
Test before production - Especially when migrating storage systems
Monitor PVC status - Alert on Pending state
Document requirements - Make access mode needs explicit
Know your cloud provider - Different services have different capabilities

🚀 Next Steps¶

Now that you understand access modes, you're ready for:

Level 34: StatefulSet volumeClaimTemplates (per-pod storage)
Level 35: StorageClass configuration and dynamic provisioning
Level 36: ConfigMap volumes and key management

📚 Additional Resources¶

Storage Capabilities by Provider: - AWS Storage Options - Google Cloud Storage - Azure Storage

Kubernetes Documentation: - Access Modes - Volume Mode

Well done! You've mastered PV/PVC access modes. Remember: choose the right access mode for your application's needs! 🎉