Skip to main content
OpenEBS logo
Star

OpenEBS for CockroachDB

EditGitHub
2.12.x

OpenEBS and CockroachDB

Introduction#

CockroachDB is a cloud-native SQL database for building global, scalable cloud services that survive disasters. It is a distributed SQL database built on a transactional and strongly consistent key-value store. It scales horizontally; survives disk, machine, rack, and even data center failures with minimal latency disruption and no manual intervention; supports strongly consistent ACID transactions; and provides a familiar SQL API for structuring, manipulating, and querying data.

This guide explains the basic installation for CockroachDB operators on OpenEBS Local PV devices.

Deployment model#

OpenEBS and CockroachDB

The Local PV volume will be provisioned on a node where CockroachDB pods are getting scheduled and uses one of the matching unclaimed block devices for each of them, which will then use the entire block device for storing data. No other application can use this device. If users have limited block devices attached to some nodes, they can use nodeSelector in the application YAML to provision applications on particular nodes where the available block device is present. The recommended configuration is to have at least three nodes and one unclaimed external disk to be attached per node.

Configuration workflow#

  1. Install OpenEBS
  2. Select OpenEBS storage engine
  3. Configure OpenEBS Local PV StorageClass
  4. Install CockroachDB Operator
  5. Accessing CockroachDB

Install OpenEBS#

If OpenEBS is not installed in your K8s cluster, this can be done from here. If OpenEBS is already installed, go to the next step.

Select OpenEBS storage engine#

A storage engine is the data plane component of the IO path of a Persistent Volume. In CAS architecture, users can choose different data planes for different application workloads based on a configuration policy. OpenEBS provides different types of storage engines and you should choose the right engine that suits your application requirements and storage available on your Kubernetes nodes. For more information you can read here.

After OpenEBS installation, choose the OpenEBS storage engine as per your requirement.

  • Choose cStor, If you are looking for replicated storage features and other enterprise graded features such as volume expansion, backup and restore, etc.

  • Choose OpenEBS Local PV, If you are looking for direct-attached storage or low latency data write or if the application manages data replication.

In this document, we are deploying CockroachDB using OpenEBS Local PV.

Configure OpenEBS Local PV StorageClass#

Depending on the type of storage attached to your Kubernetes worker nodes, you can select from different flavors of Dynamic Local PV - Hostpath, Device, LVM, ZFS, or Rawfile. For more information, you can read here.

The Storage Class openebs-device will be used to deploy CockroachDB in the Kubernetes cluster.

Note: Ensure that you have at least one disk with the required capacity added to the corresponding nodes before CockroachDB installation. In this example, we have added one 100GB disk to each node.

Install CockroachDB Operator#

  • Install the CRD using the following command.

    kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master/config/crd/bases/crdb.cockroachlabs.com_crdbclusters.yaml
    #sample output
    customresourcedefinition.apiextensions.k8s.io/crdbclusters.crdb.cockroachlabs.com created

    Note: On GKE, additional RBAC policy of cluster-admin is required. Use the following command to create it.

    kubectl create clusterrolebinding $USER-cluster-admin-binding \
    --clusterrole=cluster-admin \
    --user=<google-cloud-user-account>

    You can obtain the Google cloud user account by using the following command

    gcloud info | grep Account
    Account: [username@mayadata.io]
    #sample output
    clusterrolebinding.rbac.authorization.k8s.io/k8s-cluster-admin-binding created

  • Deploy the cockroachdb operator using the following command

    kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master/manifests/operator.yaml
    #sample output
    clusterrole.rbac.authorization.k8s.io/cockroach-database-role created
    serviceaccount/cockroach-database-sa created
    clusterrolebinding.rbac.authorization.k8s.io/cockroach-database-rolebinding created
    role.rbac.authorization.k8s.io/cockroach-operator-role created
    clusterrolebinding.rbac.authorization.k8s.io/cockroach-operator-rolebinding created
    clusterrole.rbac.authorization.k8s.io/cockroach-operator-role created
    serviceaccount/cockroach-operator-sa created
    rolebinding.rbac.authorization.k8s.io/cockroach-operator-default created
    deployment.apps/cockroach-operator created

  • Check Operator deployment pod status

    kubectl get pods
    #sample out
    NAME READY STATUS RESTARTS AGE
    cockroach-operator-599465988d-k6ffx 1/1 Running 0 48s

Deploying CockroachDB cluster#

  • Download the cluster configuration file and make the necessary changes as per your requirement.

    curl -O https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master/examples/example.yaml

    We will update the storage class to use openebs-device, as shown below. Please note that for the production environment, make necessary other changes as per your requirement.

    Sample example.yaml changes

    ---
    apiVersion: crdb.cockroachlabs.com/v1alpha1
    kind: CrdbCluster
    metadata:
    name: cockroachdb
    spec:
    dataStore:
    pvc:
    spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: "60Gi"
    volumeMode: Filesystem
    storageClassName: openebs-device
    tlsEnabled: true
    image:
    name: cockroachdb/cockroach:v20.2.5
    nodes: 3

  • Apply the cluster configuration file

    kubectl apply -f example.yaml

Verify CockroachDB is up and running#

  • Get the CockroachDB Pods, StatefulSet, Service, and PVC details. It should show that StatefulSet is deployed with 3 cockroach pods in a running state.

    kubectl get pod,pv,pvc,sc
    #sample output
    NAME READY STATUS RESTARTS AGE
    pod/cockroach-operator-599465988d-fkgv6 1/1 Running 0 5m20s
    pod/cockroachdb-0 1/1 Running 0 2m17s
    pod/cockroachdb-1 1/1 Running 0 110s
    pod/cockroachdb-2 1/1 Running 0 81s
    NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
    persistentvolume/pvc-6f0a99a2-504a-4ab7-b865-200f96bfc6cb 60Gi RWO Delete Bound default/datadir-cockroachdb-1 openebs-device 104s
    persistentvolume/pvc-a71b5078-f56f-4e1f-9237-43cfd854195e 60Gi RWO Delete Bound default/datadir-cockroachdb-0 openebs-device 2m12s
    persistentvolume/pvc-de6ec858-0106-4454-8190-66cd2a9b465f 60Gi RWO Delete Bound default/datadir-cockroachdb-2 openebs-device 76s
    NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
    persistentvolumeclaim/datadir-cockroachdb-0 Bound pvc-a71b5078-f56f-4e1f-9237-43cfd854195e 60Gi RWO openebs-device 2m19s
    persistentvolumeclaim/datadir-cockroachdb-1 Bound pvc-6f0a99a2-504a-4ab7-b865-200f96bfc6cb 60Gi RWO openebs-device 111s
    persistentvolumeclaim/datadir-cockroachdb-2 Bound pvc-de6ec858-0106-4454-8190-66cd2a9b465f 60Gi RWO openebs-device 82s
    NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
    storageclass.storage.k8s.io/openebs-device openebs.io/local Delete WaitForFirstConsumer false 25m
    storageclass.storage.k8s.io/openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 25m
    storageclass.storage.k8s.io/openebs-jiva-default openebs.io/provisioner-iscsi Delete Immediate false 25m
    storageclass.storage.k8s.io/openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter Delete Immediate false 25m
    storageclass.storage.k8s.io/premium-rwo pd.csi.storage.gke.io Delete WaitForFirstConsumer true 38m
    storageclass.storage.k8s.io/standard (default) kubernetes.io/gce-pd Delete Immediate true 38m
    storageclass.storage.k8s.io/standard-rwo pd.csi.storage.gke.io Delete WaitForFirstConsumer true 38m

Accessing CockroachDB#

  • We will be using the built-in sql-client for accessing and running some sql queries. Enter into one of the cockroachdb pods by using the exec command.

    kubectl exec -it cockroachdb-2 -- ./cockroach sql --certs-dir cockroach-certs
    #sample output
    #
    # Welcome to the CockroachDB SQL shell.
    # All statements must be terminated by a semicolon.
    # To exit, type: \q.
    #
    # Server version: CockroachDB CCL v20.2.5 (x86_64-unknown-linux-gnu, built 2021/02/16 12:52:58, go1.13.14) (same version as client)
    # Cluster ID: e51bfde5-2e75-4991-844e-d769f4b9b684
    #
    # Enter \? for a brief introduction.
    #
    root@:26257/defaultdb>

  • Run some basic SQL queries. Example command:

    root@:26257/defaultdb> CREATE DATABASE bank;
    root@:26257/defaultdb> CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL);
    root@:26257/defaultdb> INSERT INTO bank.accounts VALUES (1, 1000.50);
    root@:26257/defaultdb> SELECT * FROM bank.accounts;
    id | balance
    -----+----------
    1 | 1000.50
    (1 row)

  • Create a database user with a password for accessing the database using web UI.

    root@:26257/defaultdb> CREATE USER roach WITH PASSWORD 'Q7gc8rEdS';
    root@:26257/defaultdb> GRANT admin TO roach;

  • Create one more database, which will be used later for running benchmark load

    root@:26257/defaultdb> CREATE DATABASE sbtest;
    root@:26257/defaultdb> \q

Accessing the database's web interface.#

We will be using NodePort for accessing the service. In the production environment either use a load balancer or ingress services as per your requirement

  • Create a new node port service using the following YAML spec. The following sample spec is saved as cockroachdb-public-node-port.yaml.

    apiVersion: v1
    kind: Service
    metadata:
    name: cockroachdb-public-nodeport
    namespace: default
    spec:
    ports:
    - name: grpc
    port: 26257
    protocol: TCP
    targetPort: 26257
    - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
    selector:
    app.kubernetes.io/component: database
    app.kubernetes.io/instance: cockroachdb
    app.kubernetes.io/name: cockroachdb
    sessionAffinity: None
    type: NodePort
    kubectl apply -f cockroachdb-public-node-port.yaml

  • Get the service details

    kubectl get svc
    #sample output
    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    cockroachdb ClusterIP None <none> 26257/TCP,8080/TCP 6m57s
    cockroachdb-public ClusterIP 10.68.5.179 <none> 26257/TCP,8080/TCP 6m57s
    cockroachdb-public-nodeport NodePort 10.68.4.195 <none> 26257:30324/TCP,8080:31937/TCP 5s
    kubernetes ClusterIP 10.68.0.1 <none> 443/TCP 43m

  • Verify that the cockroachDB Dashboard is accessible using web interface

    https://<any_node_external-ip>:<NodePort>

    Login credentials for the web UI Username: roach Password: Q7gc8rEdS

See Also:#

OpenEBS architecture OpenEBS use cases Local PV concepts Understanding NDM

Was this page helpful? We appreciate your feedback