Knowledge Base

How do I reuse an existing PV - after re-creating Kubernetes StatefulSet and its PVC

There are some cases where it had to delete the StatefulSet and re-install a new StatefulSet. In the process you may have to delete the PVCs used by the StatefulSet and retain PV policy by ensuring the Retain as the "Reclaim Policy". In this case, following are the procedures for re-using an existing PV in your StatefulSet application.

1. Get the PV name by following command and use it in Step 2.

   kubectl get pv

   Following is an example output

   NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE

   pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7 5G RWO Delete Bound default/mongo-persistent-storage-mongo-0 mongo-pv-az 9m

2. Patch corresponding PV's reclaim policy from "Delete" to "Retain". So that PV will retain even its PVC is deleted.This can be done by using the steps mentioned here.

   Example Output:

   NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE

   pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7 5G RWO Retain Bound default/mongo-persistent-storage-mongo-0 mongo-pv-az 9m

3. Get the PVC name by following command and note down the PVC name. You have to use this same PVC name while creating new PVC.

   kubectl get pvc

   Example Output:

   NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

   mongo-persistent-storage-mongo-0 Lost pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7 0 mongo-pv-az 4s

4. Delete StatefulSet application and associated PVCs.

5. Create a new PVC YAML named newPVC.yaml with same configuration. Specify old PV name belongs to volumeName under the PVC spec.

   apiVersion: v1

   kind: PersistentVolumeClaim

   metadata:

   annotations:

   pv.kubernetes.io/bind-completed: "yes"

   pv.kubernetes.io/bound-by-controller: "yes"

   volume.beta.kubernetes.io/storage-provisioner: openebs.io/provisioner-iscsi

   labels:

   environment: test

   openebs.io/replica-anti-affinity: vehicle-db

   role: mongo

   name: mongo-persistent-storage-mongo-0

   namespace: default

   spec:

   accessModes:

   - ReadWriteOnce

   resources:

   requests:

   storage: 5G

   storageClassName: mongo-pv-az

   volumeName: pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7

   status:

   accessModes:

   - ReadWriteOnce

   capacity:

   storage: 5G

6. Apply the modified PVC YAML using the following command

   kubectl apply -f newPVC.yaml

   Example Output:

   NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

   mongo-persistent-storage-mongo-0 Lost pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7 0 mongo-pv-az 4s

7. Get the newly created PVC UID using kubectl get pvc mongo-persistent-storage-mongo-0 -o yaml.

8. Update the uid under the claimRef in the PV using the following command. The PVC will get attached to the PV after editing the PV with correct uid.

   kubectl edit pv pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7

9. Get the updated PVC status using the following command.

   kubectl get pvc

   Example Output:

   NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

   mongo-persistent-storage-mongo-0 Bound pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7 5G RWO mongo-pv-az 5m

10. Apply the same StatefulSet application YAML. The pod will come back online by re-using the existing PVC. The application pod status can be get by following command.

    kubectl get pods -n <namespace>

Go to top

How to prevent container logs from exhausting disk space?

Container logs, if left unchecked, can eat into the underlying disk space causing disk-pressure conditions leading to eviction of pods running on a given node. This can be prevented by performing log-rotation based on file-size while specifying retention count. One recommended way to do this is by configuring the docker logging driver on the individual cluster nodes. Follow the steps below to enable log-rotation.

1. Configure the docker configuration file /etc/docker/daemon.json (create one if not already found) with the log-options similar to ones shown below (with desired driver, size at which logs are rotated, maximum logfile retention count and compression respectively):

   {

   "log-driver": "json-file",

   "log-opts": {

   "max-size": "400k",

   "max-file": "3",

   "compress": "true"

   }

   }

2. Restart the docker daemon on the nodes. This may cause a temporary disruption of the running containers and cause the node to show up as Not Ready until the daemon has restarted successfully.

   systemctl daemon-reload

   systemctl restart docker

3. To verify that the newly set log-options have taken effect, the following commands can be used:

   • At a node-level, the docker logging driver in use can be checked via the following command:

     docker info

     The LogConfig section of the output must show the desired values:

     "LogConfig": {

     "Type": "json-file",

     "Config": {}

   • At the individual container level, the log options in use can be checked via the following command:

     docker inspect <container-id>

     The LogConfig section of the output must show the desired values:

     "LogConfig": {

     "Type": "json-file",

     "Config": {

     "max-file": "3",

     "max-size": "400k",

     "compress": "true"

     }

     }

4. To view the current and compressed files, check the contents of the /var/lib/docker/containers/<container-id>/ directory. The symlinks at /var/log/containers/<container-name> refer to the above.

note

• The steps are common for Linux distributions (tested on CentOS, RHEL, Ubuntu).

• Log rotation via the specified procedure is supported by docker logging driver types: json-file (default), local.

• Ensure there are no dockerd cli flags specifying the --log-opts (verify via ps -aux or service definition files in /etc/init.d or /etc/systemd/system/docker.service.d). The docker daemon fails to start if an option is duplicated between the file and the flags, regardless their value.

• These log-options are applicable only to the containers created after the dockerd restart (which is automatically taken care by the kubelet).

• The kubectl log reads the uncompressed files/symlinks at /var/log/containers and thereby show rotated/rolled-over logs. If you would like to read the retained/compressed log content as well use docker log command on the nodes. Note that reading from compressed logs can cause temporary increase in CPU utilization (on account of decompression actions performed internally).

• The log-opt compress: true: is supported from Docker version: 18.04.0. The max-file and max-size opts are supported on earlier releases as well.

Go to top

How to create a BlockDeviceClaim for a particular BlockDevice?

There are certain use cases where the user does not need some of the BlockDevices discovered by OpenEBS to be used by any of the storage engines. In such scenarios, users can manually create a BlockDeviceClaim to claim that particular BlockDevice, so that it won't be used by Local PV. The following steps can be used to claim a particular BlockDevice:

1. Download the BDC CR YAML from node-disk-manager repository.

   wget https://raw.githubusercontent.com/openebs/node-disk-manager/master/deploy/crds/openebs_v1alpha1_blockdeviceclaim_cr.yaml

2. Provide the BD name of the corresponding BlockDevice which can be obtained by running kubectl get bd -n <openebs_installed_namespace>

3. Apply the modified YAML spec using the following command:

   kubectl apply -f openebs_v1alpha1_blockdeviceclaim_cr.yaml -n <openebs_installed_namespace>

   note

   The blockdeviceclaim CR should be created on the same namespace where openebs is installed.

4. Verify if particular BDC is created for the given BD cr using the following command:

   kubectl get bdc -n <openebs_installed_namespace>

Go to top

How to provision Local PV on K3OS?

K3OS can be installed on any hypervisor The procedure for deploying K3OS on VMware environment is provided in the following section. There are 3 steps for provisioning OpenEBS Local PV on K3OS.

1. Configure server(master)
2. Configure agent(worker)
3. Deploying OpenEBS

The detailed information of each steps are provided below.

• Configure server(master)

  • Download the ISO file from the latest release and create a virtual machine in VMware. Mount the ISO file into hypervisor and start a virtual machine.

  • Select Run k3OS LiveCD or Installation and press <ENTER>.

  • The system will boot-up and gives you the login prompt.

  • Login as rancher user without providing password.

  • Set a password for rancher user to enable connectivity from other machines by running sudo passwd rancher.

  • Now, install K3OS into disk. This can be done by running the command sudo os-config.

  • Choose the option 1.Install to disk . Answer the proceeding questions and provide rancher user password.

  • As part of above command execution, you can configure the host as either server or agent. Select 1.server to configure K3s master.

  • While configuring server, set cluster secret which would be used while joining nodes to the server. After successful installation and server reboot, check the cluster status.

  • Run following command to get the details of nodes:

    kubectl get nodes

    Example output:

    NAME STATUS ROLES AGE VERSION

    k3os-14539 Ready <none> 2m33s v1.14.1-k3s.4

• Configure agent(worker)

  • Follow the above steps till installing K3OS into disk in all the hosts that you want to be part of K3s cluster.

  • To configure kubernetes agent with K3OS, select the option 2. agent while running sudo os-config command. You need to provide URL of server and secret configured during server configuration.

  • After performing this, Kubernetes agent will be configured as follows and it will be added to the server.

  • Check the cluster configuration by checking the nodes using the following command:

    Kubectl get nodes

    Example output:

    NAME STATUS ROLES AGE VERSION

    k3os-14539 Ready <none> 5m16s v1.14.1-k3s.4

    k3os-32750 Ready <none> 49m v1.14.1-k3s.4

• Installing OpenEBS

  • Run the following command to install OpenEBS from master console:

    kubectl apply -f https://openebs.github.io/charts/openebs-operator-1.1.0.yaml

  • Check the OpenEBS components by running the following command:

    NAME READY STATUS RESTARTS AGE

    maya-apiserver-78c966c446-zpvhh 1/1 Running 2 101s

    openebs-admission-server-66f46564f5-8sz8c 1/1 Running 0 101s

    openebs-localpv-provisioner-698496cf9b-wkf95 1/1 Running 0 101s

    openebs-ndm-9kt4n 0/1 ContainerCreating 0 101s

    openebs-ndm-mxqcf 0/1 ContainerCreating 0 101s

    openebs-ndm-operator-7fb4894546-d2whz 1/1 Running 1 101s

    openebs-provisioner-7f9c99cf9-9jlgc 1/1 Running 0 101s

    openebs-snapshot-operator-79f7d56c7d-tk24k 2/2 Running 0 101s

    note

    openebs-ndm pods are in not created successfully. This is due to the lack of udev support in K3OS. More details can be found here.

  • Now user can install Local PV on this cluster. Check the StorageClasses created as part of OpenEBS deployment by running the following command.

    kubectl get sc

    Example output:

    NAME PROVISIONER AGE

    openebs-hostpath openebs.io/local 57m

    openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter 57m

  • The default StorageClass openebs-hostpath can be used to create local PV on the path /var/openebs/local in your Kubernetes node. You can either use openebs-hostpath storage class to create volumes or create new storage class by following the steps mentioned here.

    note

    OpenEBS local PV will not be bound until the application pod is scheduled as its volumeBindingMode is set to WaitForFirstConsumer. Once the application pod is scheduled on a certain node, OpenEBS Local PV will be bound on that node.

Go to top