cStor is one of the storage engines provided by OpenEBS. The integral component of the cStor engine is its storage pool from which the volumes are created. The storage pool is constructed with the collection of block devices. When the pool is completely utilized, it may misbehave in such a way that the pool itself cannot be imported successfully to recover from failures.
In order to overcome this situation, cStor recommends the optimal usage of storage capacity in the pool by restricting the write I/Os by converting it into read-only when the threshold limit is exceeded. While creating cStor SPC, the field roThresholdLimit has to be specified in percentage value under pool spec as follows:
apiVersion: openebs.io/v1alpha1 kind: StoragePoolClaim metadata: name: cstor-pool annotations: cas.openebs.io/config: | - name: PoolResourceRequests value: |- memory: 2Gi - name: PoolResourceLimits value: |- memory: 4Gi spec: name: cstor-pool type: disk poolSpec: poolType: striped roThresholdLimit: 80 blockDevices: blockDeviceList: - blockdevice-31e0768585cb80ed2352affa73ec94e2 - blockdevice-ab636ddeba8f8cd45f7e91a6b55c15e5 - blockdevice-75275112e966e43c2ac1311a7a492fac
In the above snippet, roThresholdLimit: 80 indicates that the pool will become read-only when the usage exceeds 80% of its total capacity. Upon trying to create the above SPC, the following CSPs will be created.
NAME ALLOCATED FREE CAPACITY STATUS READONLY TYPE AGE cstor-pool-0vl0 45G 4.7G 49.8G Healthy true striped 2m8s cstor-pool-qnm1 77K 49.7G 49.8G Healthy false striped 2m8s cstor-pool-x4gj 77K 49.7G 49.8G Healthy false striped 2m8s
The read-only status of each CSP is indicated, as shown in the above template.
When the usage of the pool crosses 80% of its actual capacity, it will become read-only. It means all the replicas in that pool won’t serve further write IOs irrespective of the amount of space they consumed. As an impact, if the persistent volumes do not have enough healthy storage replicas, then the volume will become read-only.
The status of active replica where write IOs were happening turns offline when the pool becomes read-only whereas, for the idle replica, it remains healthy. The sample output of CVR is as below. Here, one replica is in Offline state as its pool is in a read-only state.
k8s@master:kubectl get cvr -n openebs NAME USED ALLOCATED STATUS AGE pvc-261d6832-8b23-476c-8aa3-b95104e20030-cstor-pool-0vl0 1.15G 1.04G Offline 23m pvc-f38f5517-a7bc-492d-a6eb-27ac510ced3b-cstor-pool-qnm1 74.7K 17.7K Healthy 23m56s
The roThresholdLimit is the configurable value. In case, if you want to increase/decrease the percentage value, it has to be modified at each CSP level. Editing at SPC config won’t be effective. In case if the administrator didn’t set the roThresholdLimit field during SPC creation, the default value would be set to, 85% considering it as the optimal value for usage. Administrators can set the roThresholdLimit value in CSP from 0 to 100 though the OpenEBS team won't recommend setting it to 100 percent.
When the pool became read-only, the administrator can either increase the pool capacity by executing the steps specified here or increase the roThresholdLimit value in that CSP to make pool RW.