OpenEBS for MinIO

This is the documentation for OpenEBS Docs 3.10.x, which is no longer actively maintained.

See the latest version (4.1.x) to view the latest documentation.

OpenEBS and MinIO

Introduction#

MinIO is a high performance distributed object storage server, designed for large-scale private cloud infrastructure. MinIO is designed in a cloud-native manner to scale sustainably in multi-tenant environments. Orchestration platforms like Kubernetes provide a perfect cloud-native environment to deploy and scale MinIO.

MinIO can be provisioned with OpenEBS volumes using various OpenEBS storage engines such as Local PV, cStor, or Jiva based on the application requirement. The MinIO operator offers a seamless way to create and update highly available distributed MinIO clusters. MinIO operator brings native support for MinIO, Graphical Console for Admin and Users, and encryption to Kubernetes. It also offers MinIO tenant creation, management, upgrade, zone addition, and more.

Depending on the performance and high availability requirements of MinIO, you can select any of the storage engine to run MinIO with the following deployment options:

For optimal performance, deploy MinIO with OpenEBS Local PV.
If you would like to use storage layer capabilities like high availability, snapshots, incremental backups and restore and so forth, you can select OpenEBS cStor.

This document provides the instructions to setup MinIO operator using OpenEBS Local PV.

Deployment model#

In this tutorial, Local PV volume will be provisioned on the node where the application has scheduled and one of the unclaimed and active blockdevice available on the same node will be used to provision the MinIO Object storage. This blockdevice cannot be used by another application. If there are limited blockdevices attached to some of the nodes, then users can use nodeSelector in the application YAML to provision application on a particular node where the available blockdevice is present.

Install OpenEBS#

If OpenEBS is not installed in your K8s cluster, this can be done from here. If OpenEBS is already installed, go to the next step.

Select OpenEBS storage engine#

A storage engine is the data plane component of the IO path of a Persistent Volume. In CAS architecture, users can choose different data planes for different application workloads based on a configuration policy. OpenEBS provides different types of storage engines and chooses the right engine that suits your type of application requirements and storage available on your Kubernetes nodes. More information can be read from here.

In this document, it is mentioned about the installation of MinIO operator using OpenEBS Local PV device.

Configure OpenEBS Local PV StorageClass#

There are 2 ways to use OpenEBS Local PV.

openebs-hostpath - Using this option, it will create Kubernetes Persistent Volumes that will store the data into OS host path directory at: /var/openebs/<"minio-pv-name">/. Select this option, if you don’t have any additional block devices attached to Kubernetes nodes. You would like to customize the directory where data will be saved, create a new OpenEBS Local PV storage class using these instructions.
openebs-device - Using this option, it will create Kubernetes Local PVs using the block devices attached to the node. Select this option when you want to dedicate a complete block device on a node to a MinIO node. You can customize which devices will be discovered and managed by OpenEBS using the instructions here.

MinIO can provide the replication of data by itself in distributed mode. This method installs MinIO application, which is a StatefulSet kind. It requires a minimum of four (4) nodes to setup MinIO in distributed mode. A distributed MinIO setup with 'n' number of disks/storage has your data safe as long as n/2 or more disks/storage are online. Users should maintain a minimum (n/2 + 1) disks/storage to create new objects. So based on the requirement, the user can choose the appropriate OpenEBS storage engine to run MinIO in distributed mode. For more information on MinIO installation, see MinIO documentation.

The Storage Class openebs-device has been chosen to deploy MinIO in the Kubernetes cluster.

Install the MinIO plugin#

The MinIO operator offers MinIO Tenant (MinIO cluster) creation, management of cluster, upgrade, zone addition, and more. Install the MinIO operator plugin using the following command.

$ kubectl krew install minio

Note: Install kubectl minio plugin using krew. Installation of krew can be done from here.

Install the MinIO operator deployment#

Let’s get started by initializing the MinIO operator deployment. This is a one time process.

$ kubectl minio init

CustomResourceDefinition tenants.minio.min.io: created
ClusterRole minio-operator-role: created
ServiceAccount minio-operator: created
ClusterRoleBinding minio-operator-binding: created
MinIO Operator Deployment minio-operator: created

Verify the MinIO operator is successfully installed.

$ kubectl get pod

NAME                              READY   STATUS    RESTARTS   AGE
minio-operator-59b8965ff5-tzx8n   1/1     Running   0          18s

Install the MinIO cluster#

A tenant is a MinIO cluster created and managed by the operator. Before creating a tenant, please ensure you have requisite nodes and drives in place. In this guide, we are using 4 Nodes with one 100Gi block device attached per each node. Using the MinIO operator, the following command will generate a YAML file as per the given requirement and the file can be modified as per user specific requirements.

$ kubectl minio tenant create --name tenant1 --servers 4 --volumes 4 --capacity 400Gi -o > tenant.yaml

The above will create a YAML spec with 4 MinIO nodes with 100Gi volume. In this YAML file, we need to add the openebs-device storage class to create the 100Gi persistent volume using the device attached to each node.

Note: Ensure that the image version used for the MinIO console is 0.4.6 or higher. Otherwise, pods will be in crashloopbackoff state.

Add the following two changes to the tenant file created using the above command.

Add the following to spec.zones.volumeClaimTemplate.spec under Tenant kind.
storageClassName: openebs-device
An example snippet of the modified tenant YAML file.
serviceName: tenant1-internal-service
zones:
- resources: {}
servers: 4
volumeClaimTemplate:
apiVersion: v1
kind: persistentvolumeclaims
metadata:
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
storageClassName: openebs-device
resources:
requests:
storage: 100Gi
status: {}
volumesPerServer: 1
Also, set requestAutoCert: false so that MinIO will run in http mode. In this document, we have used http communication for accessing MinIO. The following is a sample snippet of the modified section.
mountPath: /export
requestAutoCert: false
serviceName: tenant1-internal-service

Apply the modified tenant YAML spec. The following command will install MinIO tenants under the default namespace.

$ kubectl apply -f tenant.yaml

tenant.minio.min.io/tenant1 created
secret/tenant1-creds-secret created
secret/tenant1-console-secret created

Verify the MinIO cluster creation is successfully running under the default namespace.

$ kubectl get pod

NAME                               READY   STATUS    RESTARTS   AGE
minio-operator-59b8965ff5-tzx8n    1/1     Running   0          6m46s
tenant1-console-6589f7574d-6node   1/1     Running   0          19s
tenant1-console-6589f7574d-wt47v   1/1     Running   0          19s
tenant1-zone-0-0                   1/1     Running   0          51s
tenant1-zone-0-1                   1/1     Running   0          51s
tenant1-zone-0-2                   1/1     Running   0          51s
tenant1-zone-0-3                   1/1     Running   0          50s

Verify the MinIO persistent volume details.

$ kubectl get pvc

NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
0-tenant1-zone-0-0   Bound    pvc-eff2ebdc-1658-4525-b7e2-5d57b39f144b   100Gi      RWO            openebs-device   53s
0-tenant1-zone-0-1   Bound    pvc-1a5881ae-c65a-4ebe-9233-615c6fb7f364   100Gi      RWO            openebs-device   53s
0-tenant1-zone-0-2   Bound    pvc-bd8d3521-fea9-4a48-8f66-26c6d2808997   100Gi      RWO            openebs-device   53s
0-tenant1-zone-0-3   Bound    pvc-55d6aa94-37ed-4f14-bafb-dcee1d7af9f5   100Gi      RWO            openebs-device   52s

$ kubectl get pv

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS     REASON   AGE
pvc-1a5881ae-c65a-4ebe-9233-615c6fb7f364   100Gi      RWO            Delete           Bound    default/0-tenant1-zone-0-1   openebs-device            49s
pvc-55d6aa94-37ed-4f14-bafb-dcee1d7af9f5   100Gi      RWO            Delete           Bound    default/0-tenant1-zone-0-3   openebs-device            49s
pvc-bd8d3521-fea9-4a48-8f66-26c6d2808997   100Gi      RWO            Delete           Bound    default/0-tenant1-zone-0-2   openebs-device            53s
pvc-eff2ebdc-1658-4525-b7e2-5d57b39f144b   100Gi      RWO            Delete           Bound    default/0-tenant1-zone-0-0   openebs-device            54s

Verify MinIO service status.

$ kubectl get svc

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
kubernetes        ClusterIP   10.100.0.1      <none>        443/TCP             62m
minio             ClusterIP   10.100.59.34    <none>        80/TCP              57s
tenant1-console   ClusterIP   10.100.50.135   <none>        9090/TCP,9443/TCP   25s
tenant1-hl        ClusterIP   None            <none>        9000/TCP            57s

Now, MinIO has been installed successfully on your cluster.

Note: If the user needs to access MinIO outside the network, the service type can be changed or a new service should be added to use LoadBalancer or create Ingress resources for production deployment.

For ease of simplicity in testing the deployment, we are going to use NodePort. Please be advised to consider using LoadBalancer or Ingress, instead of NodePort, for production deployment.

The minio service will allow the user to access the console, and tenant1-console will allow access to the Admin console. In this guide, we have changed the service type of the services mentioned above, and the following is the output after the modification.

$ kubectl get svc

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
kubernetes        ClusterIP   10.100.0.1      <none>        443/TCP                         64m
minio             NodePort    10.100.59.34    <none>        80:32095/TCP                    3m10s
tenant1-console   NodePort    10.100.50.135   <none>        9090:30383/TCP,9443:30194/TCP   2m38s
tenant1-hl        ClusterIP   None            <none>        9000/TCP                        3m10s

Access MinIO console#

There are 2 different console for User and Admin.

Access MinIO Admin console#

An Admin can access MinIO and do the configuration changes such as creating an account, group, bucket, and its configuration, the setting of user-level permission, file-level permission, etc.

For Admin access, use <Node_External_Ip>:<NodePort_of_tenant1-console_service> in your web browser.

Get the details of Node.

$ kubectl get node -o wide

NAME                                            STATUS   ROLES    AGE   VERSION    INTERNAL-IP      EXTERNAL-IP      OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
ip-192-168-2-55.ap-south-1.compute.internal     Ready    <none>   51m   v1.17.11   192.168.2.55     65.0.121.83      Ubuntu 18.04.5 LTS   5.4.0-1028-aws   docker://17.3.2
ip-192-168-56-236.ap-south-1.compute.internal   Ready    <none>   57m   v1.17.11   192.168.56.236   15.206.189.106   Ubuntu 18.04.5 LTS   5.4.0-1028-aws   docker://17.3.2
ip-192-168-69-83.ap-south-1.compute.internal    Ready    <none>   57m   v1.17.11   192.168.69.83    3.6.91.169       Ubuntu 18.04.5 LTS   5.4.0-1028-aws   docker://17.3.2
ip-192-168-8-117.ap-south-1.compute.internal    Ready    <none>   57m   v1.17.11   192.168.8.117    13.235.210.41    Ubuntu 18.04.5 LTS   5.4.0-1028-aws   docker://17.3.2

Now, access the MinIO service over the browser using the following way.

http://3.6.91.169:30383

Note: Ensure Inbound Rules under VPC-> Security Groups are correctly configured to allow the traffic.

You should enter the Access key and Secret key to login into the admin console. These credentials can be obtained from the secret.

$ kubectl get secret tenant1-console-secret -o yaml

The following is a sample snippet of the output of the above command. It will show the Access key and Secret key in encoded form. The decoded value should be given in the web browser to login to the user console.

apiVersion: v1
data:
  CONSOLE_ACCESS_KEY: MmRkYzA2NGItYTMwZS00ZDg5LTgwODItNWMwYzFkYTRlOGNh
  CONSOLE_HMAC_JWT_SECRET: ODkwYWFlYmEtMTAxYy00YTJmLTg0NDMtYmI1ZjAyMjcyNWFk
  CONSOLE_PBKDF_PASSPHRASE: MDZhN2UzMmUtOWIxZi00MjI2LTk2MmItOTk4OTRmMGYwYjk2
  CONSOLE_PBKDF_SALT: OTg0OTM1YjAtNzgyMS00NWI3LWFmM2ItYzczNDZlNmUzYWNm
  CONSOLE_SECRET_KEY: MGQyY2NlZjktOWM0NC00N2JjLWFkMTYtM2RlNGExMjcwMzY1
kind: Secret
metadata:
  annotations:

Decoding of the above credentials can be retrieved by following way.

Access key

$ echo 'MmRkYzA2NGItYTMwZS00ZDg5LTgwODItNWMwYzFkYTRlOGNh' | base64 -d
2ddc064b-a30e-4d89-8082-5c0c1da4e8ca

Secret key

$ echo 'MGQyY2NlZjktOWM0NC00N2JjLWFkMTYtM2RlNGExMjcwMzY1' | base64 -d
0d2ccef9-9c44-47bc-ad16-3de4a1270365

Access MinIO User console#

The MinIO StatefulSet application is created using NodePort as the service type. To access MinIO over a web browser, use <Node_External_Ip>:<NodePort_of_minio_service> this way.