Skip to content

Solution

Triage

  1. Inspect the PV status and finalizers: 
    kubectl get pv pv-data-warehouse-01 -o yaml
  2. Check for bound PVCs: 
    kubectl get pvc --all-namespaces | grep pv-data-warehouse-01
  3. Check for VolumeAttachments: 
    kubectl get volumeattachment | grep pv-data-warehouse-01
  4. Check CSI driver pod logs: 
    kubectl logs -n kube-system -l app=csi-controller --tail=50

Root Cause

The PV has a finalizer external-provisioner.volume.kubernetes.io/finalizer set by the CSI external-provisioner. This finalizer is intended to ensure the underlying storage is properly cleaned up before the PV object is removed. However, because the reclaim policy is Retain, the CSI provisioner does not delete the backing storage and does not clear the finalizer. The PV sits in Terminating indefinitely.

Additionally, a stale VolumeAttachment object may exist if the pod was force-deleted without proper detach. The CSI attacher will not clear its finalizer until the attachment is removed.

Fix

  1. Delete any stale VolumeAttachment: 

    kubectl get volumeattachment | grep pv-data-warehouse-01
kubectl delete volumeattachment <attachment-name>

  2. Remove the finalizer from the PV: 

    kubectl patch pv pv-data-warehouse-01 -p '{"metadata":{"finalizers":null}}'
    The PV will be deleted immediately after the finalizer is removed.

  3. Clean up the underlying storage manually (since reclaim policy is Retain):

  4. Verify the backing disk/volume in your cloud provider console.

  5. Delete it manually or keep it if the data is needed for archival.

  6. Create the new PV or let dynamic provisioning handle the new PVC.

Rollback / Safety

  • Before removing the finalizer, triple-check that no pod is using this volume. Removing the finalizer while a pod has the volume mounted can cause data corruption.
  • If the underlying storage needs to be preserved, document the cloud resource ID before removing the PV.
  • Removing the finalizer is irreversible: the PV will be deleted immediately.

Common Traps

  • Assuming kubectl delete pv is enough. Finalizers block deletion. The delete command marks the PV for deletion but does not remove it.
  • Ignoring VolumeAttachments. A stale VolumeAttachment can prevent both the finalizer from clearing and the underlying storage from being detached.
  • Confusing PV protection with CSI finalizers. kubernetes.io/pv-protection prevents deletion while the PV is bound. CSI finalizers prevent deletion until storage cleanup is complete.
  • Removing finalizers on PVs with Delete reclaim policy. This skips the storage cleanup, leaving orphaned cloud disks that incur cost.
  • Not checking CSI controller health. If the CSI controller pod is crashed or OOMKilled, it cannot process finalizers. Fix the controller first.