vSphere CSI Driver - v2.3.0 release

New Features

• XFS Filesystem support as an Alpha feature. Volumes can be provisioned by specifying xfs as the csi.storage.k8s.io/fstype in the Storage Class.
• Raw Block volume support as an Alpha feature. Visit the Raw Block Volumes documentation for more details.

Notable Changes

• Node Driver DaemonSet runs with hostNetwork. Refer to (#1217) for more detail.
• Ability to set the maximum volumes that can be attached to a Node VM. This can be modified in the CSI 2.3 deployment manifest.
• Upgrade CSI driver base image to Photon 4.0. (#866)

• Modifications in ControllerUnpublishVolume error handling (#840):

  • If the Node VM is deleted from the vCenter inventory, the detach operation will be marked as successful.
  • If a NotFound fault is returned by CNS, the VM devices are checked to ensure the volume is not attached to a Node VM.

• Bug fixes

  • Disable volume length check. (#801)
  • Set volume health annotation to Inaccessible if it is not set by SPBM as it implies volume does not exist. (#852)
  • Set volume health annotation to Inaccessible when PVC not found in CNS. (#944)
  • Prevent incorrect reloading of Virtual Center configuration in CSI Syncer when credentials are not changed. (#967)
  • Prevent syncer from crashing when volumes are migrated from VCP. (#1013)
  • Reset Volume Manager during reload config when user or ca-file is updated. (#1043)

Deployment files

• https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/v2.3.0/manifests/vanilla

Kubernetes Release

• Minimum: 1.19
• Maximum: 1.21

Supported sidecar containers versions

• csi-provisioner - v2.2.0
• csi-attacher - v3.2.0
• csi-resizer - v1.1.0
• livenessprobe - v2.2.0
• csi-node-driver-registrar - v2.1.0

Known Issues

vSphere CSI Driver issues

1. Migrated in-tree vSphere volume deleted by in-tree vSphere plugin remains on the CNS UI.
   • Impact: Migrated in-tree vSphere volumes deleted by in-tree vSphere plugin remains on the CNS UI.
   • Workaround: Admin needs to manually reconcile discrepancies in the Managed Virtual Disk Catalog. Admin needs to follow this KB article - https://kb.vmware.com/s/article/2147750
2. Volume expansion might fail when it is called with Pod creation simultaneously. This issue is fixed in vSphere 7.0u2 onwards, but present on vSphere 7.0u1 and 7.0.
   • Impact: Users can resize the PVC and create a Pod using that PVC simultaneously. In this case, Pod creation might be completed first using the PVC with original size. Volume expansion will fail because online resize is not supported in vSphere 7.0 Update1.
   • Workaround: Wait for the PVC to reach FileVolumeResizePending condition before attaching a Pod to it.
3. Deleting PV before deleting PVC, leaves orphan volume on the datastore.
   • Impact: Orphan volumes remain on the datastore, and admin needs to delete those volumes manually using govc command.
   • Upstream issue is tracked at: https://github.com/kubernetes-csi/external-provisioner/issues/546
   • Workaround:
     • No workaround. User should not attempt to delete PV which is bound to PVC. User should only delete a PV if they know that the underlying volume in the storage system is gone.
     • If user has accidentally left orphan volumes on the datastore by not following the guideline, and if user has captured the volume handles or First Class Disk IDs of deleted PVs, storage admin can help delete those volumes using govc disk.rm <volume-handle/FCD ID> command.
4. Deleting a standalone Pod with inline, migrated vSphere volume does not get cleaned up immediately.
   • Issue: https://github.com/kubernetes/kubernetes/issues/103745
   • Impact: Pod with inline, migrated vSphere volume remains in Terminating state in the cluster for longer duration.
   • Workaround: Wait for Pod to be garbage collected, or force delete the pod.

Kubernetes issues

1. Filesystem resize is skipped if the original PVC is deleted when FilesystemResizePending condition is still on the PVC, but PV and its associated volume on the storage system are not deleted due to the Retain policy.
   • Issue: https://github.com/kubernetes/kubernetes/issues/88683
   • Impact: User may create a new PVC to statically bind to the undeleted PV. In this case, the volume on the storage system is resized but the filesystem is not resized accordingly. User may try to write to the volume whose filesystem is out of capacity.
   • Workaround: User can log into the container to manually resize the filesystem.
2. Volume associated with a Statefulset cannot be resized
   • Issue: https://github.com/kubernetes/enhancements/pull/660
   • Impact: User cannot resize volume in a StatefulSet.
   • Workaround: If the statefulset is not managed by an operator, there is a slightly risky workaround which the user can use on their own discretion depending upon their use case. Please refer to https://serverfault.com/questions/955293/how-to-increase-disk-size-in-a-stateful-set for more details.
3. Recover from volume expansion failure.
   • Impact: If a user tries to expand a PVC to a size which may not be supported by the underlying storage system, volume expansion will keep failing and there is no way to recover.
   • Issue: https://github.com/kubernetes/enhancements/pull/1516
   • Workaround: None