Version: 4.3.x

Knowledge Base

How do I reuse an existing PV - after re-creating Kubernetes StatefulSet and its PVC

There are some cases where it had to delete the StatefulSet and re-install a new StatefulSet. In the process you may have to delete the PVCs used by the StatefulSet and retain PV policy by ensuring the Retain as the "Reclaim Policy". In this case, following are the procedures for re-using an existing PV in your StatefulSet application.

Get the PV name by following command and use it in Step 2.

kubectl get pv

Following is an example output

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                      STORAGECLASS   REASON    AGE
pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7   5G         RWO            Delete           Bound     default/mongo-persistent-storage-mongo-0   mongo-pv-az              9m

Patch corresponding PV's reclaim policy from "Delete" to "Retain". So that PV will retain even its PVC is deleted.This can be done by using the steps mentioned here.

Example Output:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                      STORAGECLASS   REASON    AGE
pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7   5G         RWO            Retain           Bound     default/mongo-persistent-storage-mongo-0   mongo-pv-az              9m

Get the PVC name by following command and note down the PVC name. You have to use this same PVC name while creating new PVC.

kubectl get pvc

Example Output:

NAME                               STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongo-persistent-storage-mongo-0   Lost      pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7   0                        mongo-pv-az    4s

Delete StatefulSet application and associated PVCs.

Create a new PVC YAML named newPVC.yaml with same configuration. Specify old PV name belongs to volumeName under the PVC spec.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: openebs.io/provisioner-iscsi
  labels:
    environment: test
    openebs.io/replica-anti-affinity: vehicle-db
    role: mongo
  name: mongo-persistent-storage-mongo-0
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5G
  storageClassName: mongo-pv-az
  volumeName: pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5G

Apply the modified PVC YAML using the following command

kubectl apply -f newPVC.yaml

Example Output:

NAME                               STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongo-persistent-storage-mongo-0   Lost      pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7   0                        mongo-pv-az    4s

Get the newly created PVC UID using kubectl get pvc mongo-persistent-storage-mongo-0 -o yaml.
Update the uid under the claimRef in the PV using the following command. The PVC will get attached to the PV after editing the PV with correct uid.
```
kubectl edit pv pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7
```

Get the updated PVC status using the following command.

kubectl get pvc

Example Output:

NAME                               STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongo-persistent-storage-mongo-0   Bound     pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7   5G         RWO            mongo-pv-az    5m

Apply the same StatefulSet application YAML. The pod will come back online by re-using the existing PVC. The application pod status can be get by following command.

kubectl get pods -n <namespace>

Go to top

How to prevent container logs from exhausting disk space?

Container logs, if left unchecked, can eat into the underlying disk space causing disk-pressure conditions leading to eviction of pods running on a given node. This can be prevented by performing log-rotation based on file-size while specifying retention count. One recommended way to do this is by configuring the docker logging driver on the individual cluster nodes. Follow the steps below to enable log-rotation.

Configure the docker configuration file /etc/docker/daemon.json (create one if not already found) with the log-options similar to ones shown below (with desired driver, size at which logs are rotated, maximum logfile retention count and compression respectively):
```
{
  "log-driver": "json-file",
  "log-opts": {
     "max-size": "400k",
     "max-file": "3",
     "compress": "true"
  }
}
```
Restart the docker daemon on the nodes. This may cause a temporary disruption of the running containers and cause the node to show up as Not Ready until the daemon has restarted successfully.
```
systemctl daemon-reload
systemctl restart docker
```
To verify that the newly set log-options have taken effect, the following commands can be used:
- At a node-level, the docker logging driver in use can be checked via the following command:
```
docker info
```
  The LogConfig section of the output must show the desired values:
```
 "LogConfig": {
 "Type": "json-file",
 "Config": {}
```
- At the individual container level, the log options in use can be checked via the following command:
```
docker inspect <container-id>
```
  The LogConfig section of the output must show the desired values:
```
 "LogConfig": {
        "Type": "json-file",
        "Config": {
            "max-file": "3",
            "max-size": "400k",
            "compress": "true"
         }
 }      
```
To view the current and compressed files, check the contents of the /var/lib/docker/containers/<container-id>/ directory. The symlinks at /var/log/containers/<container-name> refer to the above.

note

The steps are common for Linux distributions (tested on CentOS, RHEL, Ubuntu).
Log rotation via the specified procedure is supported by docker logging driver types: json-file (default), local.
Ensure there are no dockerd cli flags specifying the --log-opts (verify via ps -aux or service definition files in /etc/init.d or /etc/systemd/system/docker.service.d). The docker daemon fails to start if an option is duplicated between the file and the flags, regardless their value.
These log-options are applicable only to the containers created after the dockerd restart (which is automatically taken care by the kubelet).
The kubectl log reads the uncompressed files/symlinks at /var/log/containers and thereby show rotated/rolled-over logs. If you would like to read the retained/compressed log content as well use docker log command on the nodes. Note that reading from compressed logs can cause temporary increase in CPU utilization (on account of decompression actions performed internally).
The log-opt compress: true: is supported from Docker version: 18.04.0. The max-file and max-size opts are supported on earlier releases as well.

Go to top

How to create a BlockDeviceClaim for a particular BlockDevice?

There are certain use cases where the user does not need some of the BlockDevices discovered by OpenEBS to be used by any of the storage engines. In such scenarios, users can manually create a BlockDeviceClaim to claim that particular BlockDevice, so that it won't be used by Local PV. The following steps can be used to claim a particular BlockDevice:

Download the BDC CR YAML from node-disk-manager repository.

wget https://raw.githubusercontent.com/openebs/node-disk-manager/master/deploy/crds/openebs_v1alpha1_blockdeviceclaim_cr.yaml

Provide the BD name of the corresponding BlockDevice which can be obtained by running kubectl get bd -n <openebs_installed_namespace>
Apply the modified YAML spec using the following command:
```
kubectl apply -f openebs_v1alpha1_blockdeviceclaim_cr.yaml -n <openebs_installed_namespace>
```
note
The blockdeviceclaim CR should be created on the same namespace where openebs is installed.
Verify if particular BDC is created for the given BD cr using the following command:
```
kubectl get bdc -n <openebs_installed_namespace>
```

Go to top

How to provision Local PV on K3OS?

K3OS can be installed on any hypervisor The procedure for deploying K3OS on VMware environment is provided in the following section. There are 3 steps for provisioning OpenEBS Local PV on K3OS.

Configure server(master)
Configure agent(worker)
Deploying OpenEBS

The detailed information of each steps are provided below.

Configure server(master)
- Download the ISO file from the latest release and create a virtual machine in VMware. Mount the ISO file into hypervisor and start a virtual machine.
- Select Run k3OS LiveCD or Installation and press <ENTER>.
- The system will boot-up and gives you the login prompt.
- Login as rancher user without providing password.
- Set a password for rancher user to enable connectivity from other machines by running sudo passwd rancher.
- Now, install K3OS into disk. This can be done by running the command sudo os-config.
- Choose the option 1.Install to disk . Answer the proceeding questions and provide rancher user password.
- As part of above command execution, you can configure the host as either server or agent. Select 1.server to configure K3s master.
- While configuring server, set cluster secret which would be used while joining nodes to the server. After successful installation and server reboot, check the cluster status.
- Run following command to get the details of nodes:
```
kubectl get nodes
```
  Example output:
```
NAME         STATUS   ROLES    AGE     VERSION
k3os-14539   Ready    <none>   2m33s   v1.14.1-k3s.4
```
Configure agent(worker)
- Follow the above steps till installing K3OS into disk in all the hosts that you want to be part of K3s cluster.
- To configure kubernetes agent with K3OS, select the option 2. agent while running sudo os-config command. You need to provide URL of server and secret configured during server configuration.
- After performing this, Kubernetes agent will be configured as follows and it will be added to the server.
- Check the cluster configuration by checking the nodes using the following command:
```
Kubectl get nodes
```
  Example output:
```
NAME         STATUS   ROLES    AGE     VERSION
k3os-14539   Ready    <none>   5m16s   v1.14.1-k3s.4
k3os-32750   Ready    <none>   49m     v1.14.1-k3s.4
```

Installing OpenEBS

Run the following command to install OpenEBS from master console:

kubectl apply -f https://openebs.github.io/charts/openebs-operator-1.1.0.yaml

Check the OpenEBS components by running the following command:

NAME                                           READY   STATUS              RESTARTS   AGE
maya-apiserver-78c966c446-zpvhh                1/1     Running             2          101s
openebs-admission-server-66f46564f5-8sz8c      1/1     Running             0          101s
openebs-localpv-provisioner-698496cf9b-wkf95   1/1     Running             0          101s
openebs-ndm-9kt4n                              0/1     ContainerCreating   0          101s
openebs-ndm-mxqcf                              0/1     ContainerCreating   0          101s
openebs-ndm-operator-7fb4894546-d2whz          1/1     Running             1          101s
openebs-provisioner-7f9c99cf9-9jlgc            1/1     Running             0          101s
openebs-snapshot-operator-79f7d56c7d-tk24k     2/2     Running             0          101s

note

openebs-ndm pods are in not created successfully. This is due to the lack of udev support in K3OS. More details can be found here.

Now user can install Local PV on this cluster. Check the StorageClasses created as part of OpenEBS deployment by running the following command.

kubectl get sc

Example output:

NAME                        PROVISIONER                                                AGE
openebs-hostpath            openebs.io/local                                           57m
openebs-snapshot-promoter   volumesnapshot.external-storage.k8s.io/snapshot-promoter   57m

The default StorageClass openebs-hostpath can be used to create local PV on the path /var/openebs/local in your Kubernetes node. You can either use openebs-hostpath storage class to create volumes or create new storage class by following the steps mentioned in the Create StorageClass(s) Documentation.

note
OpenEBS local PV will not be bound until the application pod is scheduled as its volumeBindingMode is set to WaitForFirstConsumer. Once the application pod is scheduled on a certain node, OpenEBS Local PV will be bound on that node.

Go to top

How do I reuse an existing PV - after re-creating Kubernetes StatefulSet and its PVC​

How to prevent container logs from exhausting disk space?​

How to create a BlockDeviceClaim for a particular BlockDevice?​

How to provision Local PV on K3OS?​

How do I reuse an existing PV - after re-creating Kubernetes StatefulSet and its PVC

How to prevent container logs from exhausting disk space?

How to create a BlockDeviceClaim for a particular BlockDevice?

How to provision Local PV on K3OS?