Troubleshooting OpenEBS - Dynamic LocalPV

Edit

This is the documentation for OpenEBS Docs 3.10.x, which is no longer actively maintained.

See the latest version (4.1.x) to view the latest documentation.

General guidelines for troubleshooting#

Contact OpenEBS Community for support.
Search for similar issues added in this troubleshooting section.
Search for any reported issues on StackOverflow under OpenEBS tag

LocalPV PVC in Pending state

Application pod using LocalPV device not coming into running state

Stale BDC in pending state after PVC is deleted

BDC created by localPV in pending state

PVC in Pending state#

Created a PVC using localpv-device / localpv-hostpath storage class. But the PV is not created and PVC in Pending state.

Troubleshooting: The default localpv storage classes from openebs have volumeBindingMode: WaitForFirstConsumer. This means that only when the application pod that uses the PVC is scheduled to a node, the provisioner will receive the volume provision request and will create the volume.

Resolution: Deploy an application that uses the PVC and the PV will be created and application will start using the PV

Application pod using LocalPV not coming into running state#

Application pod that uses localpv device is stuck in Pending state with error

Warning  FailedScheduling  7m24s (x2 over 7m24s)  default-scheduler  persistentvolumeclaim "<pvc-name>" not found

Troubleshooting: Check if there is a blockdevice present on the node (to which the application pod was scheduled,) which matches the capacity requirements of the PVC.

kubectl get bd -n openebs -o wide

If matching blockdevices are not present, then the PVC will never get Bound.

Resolution: Schedule the application pod to a node which has a matching blockdevice available on it.

Stale BDC in pending state after PVC is deleted#

kubectl get bdc -n openebs

shows stale Pending BDCs created by localpv provisioner, even after the corresponding PVC has been deleted.

Resolution: LocalPV provisioner currently does not delete BDCs in Pending state if the corresponding PVCs are deleted. To remove the stale BDC entries,

Edit the BDC and remove the - local.openebs.io/finalizer finalizer

kubectl edit bdc <bdc-name> -n openebs

Delete the BDC

kubectl delete bdc <bdc-name> -n openebs

BDC created by localPV in pending state#

The BDC created by localpv provisioner (bdc-pvc-xxxx) remains in pending state and PVC does not get Bound

Troubleshooting: Describe the BDC to check the events recorded on the resource

kubectl describe bdc bdc-pvc-xxxx -n openebs

The following are different types of messages shown when the node on which localpv application pod is scheduled, does not have a blockdevice available.

No blockdevices found

Warning  SelectionFailed  14m (x25 over 16m)    blockdeviceclaim-operator  no blockdevices found

It means that there were no matching blockdevices after listing based on the labels. Check if there is any block-device-tag on the storage class and corresponding tags are available on the blockdevices also

No devices with matching criteria

Warning  SelectionFailed  6m25s (x18 over 11m)  blockdeviceclaim-operator  no devices found matching the criteria

It means that the there are no devices for claiming after filtering based on filesystem type and node name. Make sure the blockdevices on the node have the correct filesystem as mentioned in the storage class (default is ext4)

No devices with matching resource requirements

Warning  SelectionFailed  85s (x74 over 11m)    blockdeviceclaim-operator  could not find a device with matching resource requirements

It means that there are no devices available on the node with a matching capacity requirement.

Resolution

To schedule the application pod to a node, which has the blockdevices available, a node selector can be used on the application pod. Here the node with hostname svc1 has blockdevices available, so a node selector is used to schedule the pod to that node.

Example:

apiVersion: v1
kind: Pod
metadata:
  name: pod1
spec:
  volumes:
    - name: local-storage
      persistentVolumeClaim:
        claimName: pvc1
  containers:
    - name: hello-container
      image: busybox
      command:
        - sh
        - -c
        - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
      volumeMounts:
        - mountPath: /mnt/store
          name: local-storage
  nodeSelector:
    kubernetes.io/hostname: svc1