Provisioning NFS PVCs

Provisioning Network File System (NFS) Persistent Volume Claims (PVCs) involves setting up storage that can be simultaneously accessed in read-write mode by multiple nodes or pods within a Kubernetes cluster.

Block storage, such as Persistent Volumes (PVs) in Kubernetes, is generally not designed to be mounted to multiple pods simultaneously. Block storage devices are designed for exclusive access. When multiple pods attempt to read from and write to the same block storage simultaneously, it can cause conflicts.

OpenEBS Replicated PV Mayastor supports ReadWriteMany (RWX) volumes by exposing regular Replicated PV Mayastor volumes via NFSv4 servers as a pod.

Concept

Replicated PV Mayastor uses the NFS server as a deployment with the NFS CSI driver. This provides PVCs in RWX mode so that multiple applications can access the data in a shared fashion. Replicated PV Mayastor volumes are used as persistent backend storage for these NFS servers to provide a scalable and manageable RWX shared storage solution.

Requirements

NFS Server as a Pod

NFS volumes can be mounted as a PersistentVolume in Kubernetes pods. It is also possible to pre-populate a NFS volume and pods may share a NFS volume. It is beneficial when certain files must be writable between multiple pods.

NFS CSI Driver

An NFS CSI driver is a specific type of Container Storage Interface (CSI) driver that enables container orchestration systems, like Kubernetes, to manage storage using the NFS. NFS (A distributed file system protocol) allows multiple machines to share directories over a network. The NFS CSI driver facilitates the use of NFS storage by providing the necessary interface for creating, mounting, and managing NFS volumes within a containerized environment, ensuring that applications running in containers can easily access and use NFS-based storage.

CSI plugin name: nfs.csi.k8s.io. This driver requires an existing and already configured NFSv3 or NFSv4 server. It supports dynamic provisioning of Persistent Volumes via PVCs by creating a new sub-directory under the NFS server. This can be deployed using Helm. See NFS CSI driver for Kubernetes to install NFS CSI driver on a Kubernetes cluster.

Replicated PV Mayastor

Replicated PV Mayastor is a performance-optimised Container Native Storage (CNS) solution. The goal of OpenEBS is to extend Kubernetes with a declarative data plane, providing flexible persistent storage for stateful applications.

Make sure you have installed Replicated PV Mayastor before proceeding to the next step. See the Installing OpenEBS Documentation to install Replicated PV Mayastor using Helm.

Details of Setup

Setting up a Single NFS Server

1. Create a Replicated PV Mayastor Pool.

Create a Replicated PV Mayastor pool that satisfies the performance and availability requirements. See Replicated PV Mayastor Configuration Documentation for more details.

Example of a Replicated PV Mayastor Pool

Command

$ kubectl get dsp -n mayastor

Output

NAME     NODE                                       STATE     POOL_STATUS   CAPACITY      USED   AVAILABLE
pool-1   gke-gke-ssd-1-default-pool-d802bc13-3j4s   Created   Online        10724835328   0      10724835328
pool-2   gke-gke-ssd-1-default-pool-d802bc13-z9hf   Created   Online        10724835328   0      10724835328
pool-3   gke-gke-ssd-1-default-pool-d802bc13-67bv   Created   Online        10724835328   0      10724835328

2. Create a Replicated PV Mayastor Storage Class.

Create a storage class to point to the above created pool. Also, select the number of replicas and the default size of the volume. See Replicated PV Mayastor Configuration Documentation for more details.

Example of a Replicated PV Mayastor Storage Class

Command

$ kubectl get sc

Output

mayastor-2                io.openebs.csi-mayastor   Delete          Immediate              false                  24h
mayastor-3                io.openebs.csi-mayastor   Delete          Immediate              false                  27h
mayastor-etcd-localpv     openebs.io/local          Delete          WaitForFirstConsumer   false                  27h
mayastor-loki-localpv     openebs.io/local          Delete          WaitForFirstConsumer   false                  27h
mayastor-single-replica   io.openebs.csi-mayastor   Delete          Immediate              true                   27h

3. Create a namespace for deploying the NFS server.

Command

$ kubectl create ns nfs-server

Output

namespace/nfs-server created

4. Create a PVC with the desired storage for the NFS server using the Replicated PV Mayastor storage class.

Command

$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-server-claim
  namespace: nfs-server
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: mayastor-3
EOF

Output

persistentvolumeclaim/nfs-server-claim created

• Use the following command to verify the PVC creation:

Command

$ kubectl get pvc -n nfs-server

Output

NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
nfs-server-claim   Bound    pvc-3763a93c-dfcd-4be3-8255-038b44bd5432   5Gi        RWO            mayastor-3     <unset>                 8s

5. Deploy the NFS server.

   • Deployment manifest

Deploy the NFS server using the following yaml file. Use a Kubernetes deployment to create the NFS server, containing a single pod running. Deploy the server with this deployment manifest.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-server
  namespace: nfs-server
spec:
  replicas: 1
  selector:
    matchLabels:
      role: nfs-server
  template:
    metadata:
      labels:
        role: nfs-server
    spec:
      volumes:
      - name: nfs-vol
        persistentVolumeClaim:
          claimName: nfs-server-claim
      restartPolicy: Always
      containers:
      - name: nfs-server
        image: itsthenetwork/nfs-server-alpine
        env:
        - name: SHARED_DIRECTORY
          value: /nfsshare
        ports:
          - name: nfs
            containerPort: 2049
        securityContext:
          privileged: true
        volumeMounts:
          - mountPath: /nfsshare
            name: nfs-vol

• Deploy the NFS server using the above deployment manifest by using the following command:

Command

$ kubectl apply -f nfs-deploy.yaml 

Output

deployment.apps/nfs-server created

• Use the following command to verify the NFS server deployment:

Command

$ kubectl get pods -n nfs-server

Output

NAME                          READY   STATUS    RESTARTS   AGE
nfs-server-779b64df65-5pqmk   1/1     Running   0          49s

• Create a service object that provides a ClusterIP to access the NFS server by using the following command:

NFS Server Example: nfs-server.nfs-server.svc.cluster.local

YAML

apiVersion: v1
kind: Service
metadata:
  name: nfs-server
  namespace: nfs-server
spec:
  ports:
    - name: nfs
      port: 2049
  selector:
    role: nfs-server

Command

$ kubectl apply -f nfs-svc.yaml

Output

service/nfs-server created

• Use the following command to verify the service object:

Command

$ kubectl get svc -n nfs-server

Output

NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
nfs-server   ClusterIP   34.118.238.249   <none>        2049/TCP   8s

Setting up a NFS CSI Driver

1. Create a storage class.

   • Edit the values.yaml to use the NFS server.

StorageClass Resource Example

storageClass:
  create: true
  name: nfs-csi
  parameters:
    server: nfs-server.nfs-server.svc.cluster.local
    share: /
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
  mountOptions:
    - nfsvers=4.1

• Create a new namespace for the NFS CSI Driver.

Command

$ kubectl create ns csi-nfs

Output

namespace/csi-nfs created

• Install NFS CSI Driver using helm.

Command

$ helm install -n csi-nfs  csi-nfs

Output

NAME: csi-nfs
LAST DEPLOYED: Wed Jun 19 15:39:25 2024
NAMESPACE: csi-nfs
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The CSI NFS Driver is getting deployed to your cluster.

• Verify the CSI NFS Driver pods status.

kubectl --namespace=csi-nfs get pods --selector="app.kubernetes.io/instance=csi-nfs" --watch

• Verify the pod status.

Command

$ kubectl get pods -n csi-nfs 

Output

NAME                                 READY   STATUS    RESTARTS   AGE
csi-nfs-controller-7fff64574-cnwdp   4/4     Running   0          53s
csi-nfs-node-j27w7                   3/3     Running   0          53s
csi-nfs-node-slp95                   3/3     Running   0          53s
csi-nfs-node-srsbm                   3/3     Running   0          53s

• Verify the storage class.

Command

$ kubectl get sc | grep nfs-csi

Output

nfs-csi                   nfs.csi.k8s.io            Delete          Immediate              false                  2m29s

Command

$ kubectl describe sc nfs-csi

Output

Name:                  nfs-csi
IsDefaultClass:        No
Annotations:           meta.helm.sh/release-name=csi-nfs,meta.helm.sh/release-namespace=csi-nfs
Provisioner:           nfs.csi.k8s.io
Parameters:            mountPermissions=0,server=nfs-server.nfs-server.svc.cluster.local,share=/
AllowVolumeExpansion:  <unset>
MountOptions:
  nfsvers=4.1
ReclaimPolicy:      Delete
VolumeBindingMode:  Immediate
Events:             <none>

NFS PVC Creation

1. Create PVC with NFS CSI Driver Storage Class nfs-csi (that was created in the last step).

YAML

$ cat nfs-pvc.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-claim-1
spec:
  storageClassName: nfs-csi
  accessModes: [ "ReadWriteMany" ]
  resources:
    requests:
       # This is the storage capacity that will be available to the application pods.
      storage: 5Gi

Command

$ kubectl apply -f nfs-pvc.yaml 

Output

persistentvolumeclaim/nfs-claim-1 created

• Use the following command to verify the PVC:

Command

$ kubectl get pvc 

Output

NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
nfs-claim-1   Bound    pvc-f795f8bc-58b3-40d7-b9b2-2d16c17f3d47   5Gi        RWX            nfs-csi        <unset>                 10s

2. Deploy the application.

YAML

$ cat nginx.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: "/volume"
            name: data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: nfs-claim-1

Command

$ kubectl get pods -o wide

Output

NAME                               READY   STATUS    RESTARTS   AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
nginx-deployment-6c664cc4c-5kjc8   1/1     Running   0          22s   10.32.1.32   gke-gke-ssd-1-default-pool-d802bc13-67bv   <none>           <none>
nginx-deployment-6c664cc4c-xzlwq   1/1     Running   0          22s   10.32.2.17   gke-gke-ssd-1-default-pool-d802bc13-z9hf   <none>           <none>
nginx-deployment-6c664cc4c-z64h4   1/1     Running   0          22s   10.32.0.14   gke-gke-ssd-1-default-pool-d802bc13-3j4s   <none>           <none>

See Also

• Replicated PV Mayastor Installation on MicroK8s
• Replicated PV Mayastor Installation on Talos
• Replicated PV Mayastor Installation on Google Kubernetes Engine