Skip to main content

OpenEBS and Elasticsearch


EFK is the most popular cloud native logging solution on Kubernetes for On-Premise as well as cloud platforms. In the EFK stack, Elasticsearch is a stateful application that needs persistent storage. Logs of production applications need to be stored for a long time which requires reliable and highly available storage. OpenEBS and EFK together provides a complete logging solution.

This guide explains the basic installation for Elasticsearch operators on OpenEBS Local PV devices using KUDO. We will be installing Fluentd and Kibana to form the EFK stack.

Advantages of using OpenEBS LocalPV for Elasticsearch database:

  • All the logs data is stored locally and managed natively to Kubernetes
  • Low latency and better performance

Deployment model#

OpenEBS and Elasticsearch

The Local PV volume will be provisioned on a node where Elasticsearch components are getting scheduled and uses one of the matching unclaimed block device for each of them, which will then use the entire block device for storing data. No other application can use this device. If users have limited blockdevices attached to some nodes, they can use nodeSelector in the application YAML to provision applications on particular nodes where the available block device is present. The recommended configuration is to have at least three nodes and two unclaimed external disk to be attached per node.

The Elasticsearch deployment has the following components, which will use the OpenEBS LocalPV Devices for storage:

  1. coordinator pod: 1
  2. master pods: 3
  3. data pods: 2

Note: Elasticsearch can be deployed both as deployment or as statefulset. When Elasticsearch deployed as statefulset, you don't need to replicate the data again at OpenEBS level. When Elasticsearch is deployed as deployment, consider 3 OpenEBS replicas, choose the StorageClass accordingly.

Configuration workflow#

  1. Install OpenEBS
  2. Select OpenEBS storage engine
  3. Configure OpenEBS Local PV StorageClass
  4. Installing KUDO Operator
  5. Installing and Accessing Elasticsearch
  6. Installing Kibana
  7. Installing Fluentd-ES

Install OpenEBS#

If OpenEBS is not installed in your K8s cluster, this can be done from here. If OpenEBS is already installed, go to the next step.

Select OpenEBS storage engine#

A storage engine is the data plane component of the IO path of a Persistent Volume. In CAS architecture, users can choose different data planes for different application workloads based on a configuration policy. OpenEBS provides different types of storage engines and you should choose the right engine that suits your type of application requirements and storage available on your Kubernetes nodes. More information can be read from here.

After OpenEBS installation, choose the OpenEBS storage engine as per your requirement.

  • Choose cStor, If you are looking for replicated storage feature and other enterprise graded features such as volume expansion, backup and restore, etc. The steps for Elasticsearch installation using OpenEBS cStor storage engine can be obtained from here.

  • Choose OpenEBS Local PV, If you are looking for direct attached storage or low latency data write or if the application manages data replication.

In this document, we are deploying Elasticsearch using OpenEBS Local PV.

Configure OpenEBS Local PV StorageClass#

Depending on the type of storage attached to your Kubernetes worker nodes, you can select from different flavors of Dynamic Local PV - Hostpath, Device, LVM, ZFS or Rawfile. For more information you can read here.

The Storage Class openebs-device has been chosen to deploy Elasticsearch in the Kubernetes cluster.

Note: Ensure that you have two disks with the required capacity added to the corresponding nodes prior to Elasticsearch installation. In this example, we have added two 100G disks to each node.

Installing KUDO Operator#

In this section, we will install the KUDO operator. We will later deploy the latest available version of Elasticsearch using KUDO.

Use the latest stable version of KUDO CLI. The latest version of KUDO can be checked from here.

Verify if Cert-manager is installed#

For installing KUDO operator, the Cert-manager must be already installed in your cluster. If not, install the Cert-manager. The instruction can be found from here. Since our K8s version is v1.18.12, we have installed Cert-manager using the following command.

kubectl apply -f
kubectl get pods --namespace cert-manager
#sample output
cert-manager-7747db9d88-7qjnm 1/1 Running 0 81m
cert-manager-cainjector-87c85c6ff-whnzr 1/1 Running 0 81m
cert-manager-webhook-64dc9fff44-qww8s 1/1 Running 0 81m

Installing KUDO operator into cluster#

Once prerequisites are installed you need to initialize the KUDO operator. The following command will install KUDO v0.18.2.

kubectl-kudo init --version 0.18.2
#sample output
$KUDO_HOME has been configured at /home/k8s/.kudo
✅ installed crds
✅ installed namespace
✅ installed service account
✅ installed webhook
✅ installed kudo controller

Verify pods in the kudo-system namespace:

kubectl get pod -n kudo-system
#sample output
kudo-controller-manager-0 1/1 Running 0 25s

Setting OpenEBS Storage Class as default#

Change the default storage class from your current setting to OpenEBS LocalPV Device. For example, in this tutorial default storage class is used as openebs-device from standard.

kubectl patch storageclass standard -p '{"metadata": {"annotations":{"":"false"}}}'
kubectl patch storageclass openebs-device -p '{"metadata": {"annotations":{"":"true"}}}'

Verify default Storage Class#

List the storage classes and verify openebs-device is set to default.

kubectl get sc
#sample output
openebs-device (default) Delete WaitForFirstConsumer false 4h30m
openebs-hostpath Delete WaitForFirstConsumer false 4h30m
openebs-jiva-default Delete Immediate false 4h30m
openebs-snapshot-promoter Delete Immediate false 4h30m
premium-rwo Delete WaitForFirstConsumer true 5h13m
standard Delete Immediate true 5h13m
standard-rwo Delete WaitForFirstConsumer true 5h13

Installing and Accessing Elasticsearch#

Set instance and namespace variables:

export instance_name=elastic
export namespace_name=default
kubectl-kudo install elastic --namespace=$namespace_name --instance $instance_name
#sample output
operator default/elastic created
operatorversion default/elastic-7.0.0-0.2.1 created
instance default/elastic created

Verifying Elastic pods#

kubectl get pods -n $namespace_name
#sample output
elastic-coordinator-0 1/1 Running 0 31s
elastic-data-0 1/1 Running 0 56s
elastic-data-1 1/1 Running 0 44s
elastic-master-0 1/1 Running 0 2m31s
elastic-master-1 1/1 Running 0 119s
elastic-master-2 1/1 Running 0 90s

Verifying Services#

kubectl get svc -n $namespace_name
#sample output
elastic-coordinator-hs ClusterIP None <none> 9200/TCP 62s
elastic-data-hs ClusterIP None <none> 9200/TCP 87s
elastic-ingest-hs ClusterIP None <none> 9200/TCP 50s
elastic-master-hs ClusterIP None <none> 9200/TCP 3m2s
kubernetes ClusterIP <none> 443/TCP 5h18m

Verifying Elastic instance status#

kubectl kudo plan status --namespace=$namespace_name \
--instance $instance_name
#sample output
Plan(s) for "elastic" in namespace "default":
└── elastic (Operator-Version: "elastic-7.0.0-0.2.1" Active-Plan: "deploy")
└── Plan deploy (serial strategy) [COMPLETE], last updated 2021-02-22 16:18:17
├── Phase deploy-master (parallel strategy) [COMPLETE]
│ └── Step deploy-master [COMPLETE]
├── Phase deploy-data (parallel strategy) [COMPLETE]
│ └── Step deploy-data [COMPLETE]
├── Phase deploy-coordinator (parallel strategy) [COMPLETE]
│ └── Step deploy-coordinator [COMPLETE]
└── Phase deploy-ingest (parallel strategy) [COMPLETE]
└── Step deploy-ingest [COMPLETE]

Accessing Elasticsearch#

Enter into one of the master pod using exec command:

kubectl exec -it elastic-master-0 -- bash
#sample output
[root@elastic-master-0 elasticsearch]#

Run below command inside Elastic master pod:

curl -X POST "elastic-coordinator-hs:9200/twitter/_doc/" -H 'Content-Type: application/json' -d'
"user" : "openebs",
"post_date" : "2021-03-02T14:12:12",
"message" : "Test data entry"

Following is the output of the above command:

#sample output
[root@elastic-master-0 elasticsearch]#

The above command added data into Elasticsearch. You can use the following command to query for the inserted data:

curl -X GET "elastic-coordinator-hs:9200/twitter/_search?q=user:openebs&pretty"
#sample output
"took" : 141,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
"max_score" : 0.2876821,
"hits" : [
"_index" : "twitter",
"_type" : "_doc",
"_id" : "qoWznXkBpvRSYN1fECvZ",
"_score" : 0.2876821,
"_source" : {
"user" : "openebs",
"post_date" : "2021-03-02T14:12:12",
"message" : "Test data entry"

Now, let's get the details of Elasticsearch cluster. The cluster information will show the Elasticsearch version, cluster name and other details. If you are getting similar information, then it means your Elasticsearch deployment is successful.

curl localhost:9200
#sample output
"name" : "elastic-master-0",
"cluster_name" : "elastic-cluster",
"cluster_uuid" : "A0qErYmCS2OpJtmgR_j3ow",
"version" : {
"number" : "7.10.1",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "1c34507e66d7db1211f66f3513706fdf548736aa",
"build_date" : "2020-12-05T01:00:33.671820Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
"tagline" : "You Know, for Search"

Installing Kibana#

First, add helm repository of Elastic.

helm repo add elastic && helm repo update

Install Kibana deployment using helm command. Ensure to meet required prerequisites corresponding to your helm version. Fetch the Kibana values.yaml:


Edit the following parameters:

  1. elasticsearchHosts as "http://elastic-coordinator-hs:9200" # service name of Elastic search.
  2. service.type as "NodePort".
  3. service.nodePort as "30295" # since this port is already added in our network firewall rules.
  4. imageTag as "7.10.1" , it should be the same image tag of Elasticsearch. In our case, Elasticsearch image tag is 7.10.1.

Now install Kibana using Helm:

helm install kibana -f values.yaml elastic/kibana

Verifying Kibana Pods and Services:

kubectl get pod
#sample output
elastic-coordinator-0 1/1 Running 0 12m
elastic-data-0 1/1 Running 0 12m
elastic-data-1 1/1 Running 0 12m
elastic-master-0 1/1 Running 0 11m
elastic-master-1 1/1 Running 0 11m
elastic-master-2 1/1 Running 0 12m
kibana-kibana-74cbc4d654-h8djr 1/1 Running 0 6m33s
kubectl get svc
#sample output
elastic-coordinator-hs ClusterIP None <none> 9200/TCP 18m
elastic-data-hs ClusterIP None <none> 9200/TCP 19m
elastic-ingest-hs ClusterIP None <none> 9200/TCP 18m
elastic-master-hs ClusterIP None <none> 9200/TCP 20m
kibana-kibana NodePort <none> 5601:30295/TCP 7m1s
kubernetes ClusterIP <none> 443/TCP 5h36m

Installing Fluentd-ES#

Fetch the values.yaml:

helm repo add bitnami && helm repo update

Replace the following section in the values.yaml file with new content.


# Send the logs to the standard output
<match **>
@type stdout


# Send the logs to the elasticsearch output
<match **>
@type elasticsearch
include_tag_key true
host elastic-coordinator-hs
port 9200
logstash_format true
@type file
path /opt/bitnami/fluentd/logs/buffers/logs.buffer
flush_thread_count 2
flush_interval 5s

Install Fluentd-Elasticsearch DaemonSet using the new values:

helm install fluentd -f values.yaml bitnami/fluentd

Verify Fluentd Daemonset, Pods and Services:

kubectl get ds
#sample output
fluentd 3 3 3 3 3 <none> 74m
kubectl get pod
#sample output
pod/elastic-coordinator-0 1/1 Running 0 67m
pod/elastic-data-0 1/1 Running 0 66m
pod/elastic-data-1 1/1 Running 0 67m
pod/elastic-master-0 1/1 Running 0 66m
pod/elastic-master-1 1/1 Running 0 66m
pod/elastic-master-2 1/1 Running 0 66m
pod/fluentd-0 1/1 Running 0 106s
pod/fluentd-4sbs4 1/1 Running 0 106s
pod/fluentd-5mvv9 1/1 Running 2 106s
pod/fluentd-z2sxt 1/1 Running 2 106s
pod/kibana-kibana-74cbc4d654-h8djr 1/1 Running 0 9m46s
kubectl get svc
#sample output
elastic-coordinator-hs ClusterIP None <none> 9200/TCP 67m
elastic-data-hs ClusterIP None <none> 9200/TCP 66m
elastic-ingest-hs ClusterIP None <none> 9200/TCP 67m
elastic-master-hs ClusterIP None <none> 9200/TCP 66m
fluentd-aggregator ClusterIP <none> 9880/TCP,24224/TCP 106s
fluentd-forwarder ClusterIP <none> 9880/TCP 106s
fluentd-headless ClusterIP None <none> 9880/TCP,24224/TCP 106s
kibana-kibana NodePort <none> 5601:30295/TCP 9m46s
kubernetes ClusterIP <none> 443/TCP 5h40m

Getting logs from the indices:

  1. Goto Kibana dashboard.
  2. Click on Management->Stack Management which is placed at left side bottom most.
  3. Click on index patterns listed under Kibana and then click on Create Index pattern.
  4. Provide logstash-* inside the index pattern box and then select Next step.
  5. In the next step, inside the Time Filter field name, select the @timestamp field from the dropdown menu, and click Create index pattern.
  6. Now click on the Discover button listed on the top left of the side menu bar.
  7. There will be a dropdown menu where you can select the available indices.
  8. In this case, you have to select logstash-* from the dropdown menu.

Now let's do some tests:

If you want to get the logs of NDM pods, type the following text inside the Filters field. kubernetes.labels.openebs_io/component-name.keyword : "ndm" and then choose the required date and time period. After that, click Apply. You will see the OpenEBS NDM pod logs listed on the page.

See Also:#

OpenEBS use cases Understanding NDM Local PV concepts Local PV User guide

Was this page helpful? We appreciate your feedback