Deploy EFK Stack on Kubernetes for Efficient Logging

Deploying an EFK Logging Stack on Kubernetes

In modern applications, logging is crucial for monitoring and debugging. The EFK stack—Elasticsearch, Fluentd, and Kibana—is a popular choice for logging and monitoring in Kubernetes environments. This guide will show you how to deploy an EFK stack on Kubernetes, allowing you to efficiently manage and visualize logs.

Prerequisites

A Kubernetes cluster up and running (e.g., Minikube, EKS, AKS, or GKE).
Kubectl CLI installed and configured to communicate with your cluster.
Basic understanding of containers and Kubernetes concepts.

Step 1: Deploy Elasticsearch

Elasticsearch serves as the backend for storing logs. We will use the Elasticsearch operator to simplify the deployment.

kubectl create -f https://download.elastic.co/downloads/eck/manifested/1.5.0/all-in-one.yaml

After deploying the operator, create an Elasticsearch cluster:


apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.9.3
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false

Apply the Elasticsearch manifest:

kubectl apply -f elasticsearch.yaml

Step 2: Deploy Fluentd

Fluentd acts as the data collector, aggregating logs across nodes and sending them to Elasticsearch.

kubectl apply -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch-rbac.yaml

Modify the Fluentd configuration as needed to capture specific log formats or add filters.

Step 3: Deploy Kibana

Kibana is the visualization layer, allowing you to query and view logs stored in Elasticsearch.


apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 7.9.3
  count: 1
  elasticsearchRef:
    name: quickstart

Apply the Kibana manifest:

kubectl apply -f kibana.yaml

Once deployed, check the Kibana service and forward the port to your localhost to access the UI. Explore your logs through the user-friendly interface.

You may refer to our previous guide on how to install Kibana on Kubernetes for further insights.

Troubleshooting

Elasticsearch not starting: Check the logs with kubectl logs [ELASTICSEARCH_POD]. Verify resources and configurations.
Fluentd not forwarding logs: Ensure Fluentd is configured correctly and reachable by Elasticsearch. Check its pod logs for errors.
Kibana connection issues: Confirm that Kibana is pointing to the correct Elasticsearch instance and network policies allow communication.

Summary

By deploying the EFK stack on Kubernetes, you create a powerful logging system that enhances your visibility into applications running within your cluster. The integration enhances real-time monitoring, troubleshooting, and system optimization.