Troubleshooting - Local Storage
#
PVC in Pending stateCreated a PVC using localpv-hostpath storage class. But the PV is not created and PVC in Pending state.
Troubleshooting
The default localpv storage classes from openebs have volumeBindingMode: WaitForFirstConsumer
. This means that only when the application pod that uses the PVC is scheduled to a node, the provisioner will receive the volume provision request and will create the volume.
Resolution
Deploy an application that uses the PVC and the PV will be created and application will start using the PV.
Installation Related#
Installation failed because of insufficient user rightsOpenEBS installation can fail in some cloud platform with the following errors.
Troubleshooting
You must enable RBAC before OpenEBS installation. This can be done from the kubernetes master console by executing the following command.
#
Why does OpenEBS provisioner pod restart continuously?The following output displays the pod status of all namespaces in which the OpenEBS provisioner is restarting continuously.
Troubleshooting
Perform the following steps to verify if the issue is due to misconfiguration while installing the network component.
Check if your network related pods are running fine.
Check if OpenEBS provisioner HTTPS requests are reaching the apiserver
Use the latest version of network provider images.
Try other network components such as Calico, kube-router etc. if you are not using any of these.
#
OpenEBS installation fails on AzureOn AKS, while installing OpenEBS using Helm, you may see the following error.
Troubleshooting
You must enable RBAC on Azure before OpenEBS installation. For more details, see Prerequisites.
#
A multipath.conf file claims all SCSI devices in OpenShiftA multipath.conf file without either find_multipaths or a manual blacklist claims all SCSI devices.
Workaround
Add the find multipaths line to _/etc/multipath.conf_ file similar to the following snippet.
Run
multipath -w /dev/sdc
command (replace the devname with your persistent devname).
#
Set Cluster-admin User ContextFor installation of OpenEBS, cluster-admin user context is a must. OpenEBS installs service accounts and custom resource definitions that are only allowed for cluster administrators.
Use the kubectl auth can-i
commands to verify that you have the cluster-admin context. You can use the following commands to verify if you have access:
If there is no cluster-admin user context already present, create one and use it. Use the following command to create the new context.
Example:
Set the existing cluster-admin user context or the newly created context by using the following command.
Example:
#
Application and OpenEBS pods terminate/restart under heavy I/O loadThis is caused due to lack of resources on the Kubernetes nodes, which causes the pods to evict under loaded conditions as the node becomes unresponsive. The pods transition from Running state to unknown state followed by Terminating before restarting again.
Troubleshooting
The above cause can be confirmed from the kubectl describe pod
which displays the termination reason as NodeControllerEviction. You can get more information from the kube-controller-manager.log on the Kubernetes master.
Workaround
You can resolve this issue by upgrading the Kubernetes cluster infrastructure resources (Memory, CPU).
Others#
Nodes in the cluster reboots frequently almost everyday in openSUSE CaaSSetup the cluster using RKE with openSUSE CaaS MicroOS using CNI Plugin Cilium. Install OpenEBS, create a PVC and allocate to a fio job/busybox. Run FIO test on the same. Observed nodes in the cluster getting restarted on a schedule basis.
Troubleshooting
Check journalctl logs of each nodes and check if similar logs are observed. In the following log snippets, showing the corresponding logs of 3 nodes.
Node1:
Node2:
Node3:
You can get more details to see if the cause of reboot is due to transactional update using below command outputs.
Workaround
There are 2 possible solutions.
Approach1:
Do the following on each nodes to stop the transactional update.
This is the preferred approach.
Approach2:
Set the reboot timer schedule at different time i.e. staggered at various interval of the day, so that only one nodes get rebooted at a time.
#
How to fetch the OpenEBS Dynamic Local Provisioner logs?Workaround
Review the logs of the OpenEBS Local PV provisioner. OpenEBS Dynamic Local Provisioner logs can be fetched using:
xfs
File System#
Unable to mount The volume is created, but xfs
is failing to mount.
Workaround
If you are trying to use xfs
volumes and the cluster node hosts are running a kernel version less than 5.10, you may encounter a mount failure of the filesystem. This is due to the incompatibility of newer xfsprogs
options. In order to alleviate this issue, it is recommended to upgrade the host node kernel version to 5.10 or higher.