Kubernetes Fixes Leaked Volumes with HonorPVReclaimPolicy

In Kubernetes 1.33, a long-standing issue that caused persistent volumes to leak under certain deletion sequences has finally been resolved. With the graduation of the HonorPVReclaimPolicy feature to GA, Kubernetes now reliably honors the reclaim policy of Persistent Volumes (PVs)—regardless of whether the PV or its associated Persistent Volume Claim (PVC) is deleted first.

The Problem: PVs Were Ignored, Volumes Were Leaked¶

Until now, reclaim policies like Delete were only honored if the PVC was deleted before the PV. In reverse order—deleting the PV first—the reclaim policy would silently fail, leading to orphaned volumes in your cloud provider or storage backend.

This behavior was deeply tied to how Kubernetes controllers interpreted object states and finalizers. If the PV disappeared from the API server before the reclaim operation was completed, the backing storage was never cleaned up.

What’s Changing: Finalizers to the Rescue¶

The fix revolves around finalizers—metadata markers that prevent deletion of a Kubernetes object until specific conditions are met.

For CSI Driver Volumes:¶

• external-provisioner.volume.kubernetes.io/finalizer is now added not just to new PVs during provisioning, but also to existing PVs.
• This prevents premature deletion from the API server.
• Once the external CSI provisioner confirms the backing volume is deleted, the finalizer is removed, and the PV is fully deleted.

For In-Tree Plugin Volumes:¶

• A new finalizer kubernetes.io/pv-controller has been introduced.
• It's added to newly created and existing PVs managed by in-tree plugins.
• The finalizer ensures the PV is only removed after a successful plugin-level volume deletion.
• It is dynamically added/removed based on CSI migration status.

How It Works Now¶

With the HonorPVReclaimPolicy feature gate enabled:

1. Delete PV first: A deletionTimestamp is added to the PV, triggering a sync loop.
2. Delete PVC next: Finalizers ensure the PV isn’t purged from the API server.
3. Controllers coordinate deletion: Either the external provisioner or in-tree controller performs the actual volume deletion.
4. Finalizer is removed: Only once the backing storage is deleted.
5. PV is finally deleted.

This applies to both dynamically and statically provisioned volumes—provided a PVC is attached.

Real-World Benefit: No More Leaked Resources¶

This enhancement:

• Closes a long-standing bug where storage resources were silently orphaned.
• Improves consistency, aligning the behavior of PVs regardless of deletion order.
• Reduces ops overhead, especially in production clusters where manual cleanup was previously required.

Enabling the Feature¶

To use this feature:

• Ensure the HonorPVReclaimPolicy feature gate is enabled on both kube-controller-manager and the external CSI provisioner.
• Supported since v1.26 (alpha), v1.31 (beta), and v1.33 (GA).

--feature-gates=HonorPVReclaimPolicy=true

Metrics and Monitoring¶

You can confirm it’s working by inspecting the PV object:

• For CSI: metadata.finalizers should contain external-provisioner.volume.kubernetes.io/finalizer.
• For in-tree: kubernetes.io/pv-controller should be present.

Metrics to monitor:

• persistentvolume_delete_failed_total
• persistentvolume_delete_duration_seconds

Edge Cases and Safeguards¶

• Statically provisioned PVs without PVCs won’t get finalizers.
• Finalizers may block deletion if CSI drivers or controllers are not correctly upgraded.
• Manual intervention may be required if you disable the feature and have lingering finalizers.

Summary¶

The HonorPVReclaimPolicy feature provides a definitive fix for volume leak issues in Kubernetes. By leveraging finalizers and refining controller coordination, it ensures that the reclaim policy is reliably honored—no matter the deletion order.

This is a major step forward for storage lifecycle management in Kubernetes, closing gaps that have persisted for years.