Install EFK Stack For Unified Logging On Kubernetes

Setting up EFK with some basic examples of usage.


Introduction

Here's a basic example of installing and using unified logging on Kubernetes. I assume you now nothing about it for this tutorial. My personal preference is to use the EFK stack, which is Elasic Search for storage and indexing, Fluentd for aggregating logs for all of your containers, and Kibana to visualize and search through your logs.

What This Tutorial Covers

What This Tutorial Covers
  1. Installing Elasic Search
  2. Installing Fluentd
  3. Installing Kibana
  4. Basic EFK Usage

What You Need For This Tutorial

What You Need For This Tutorial

A Kubernetes Cluster


Install EFK

Helm makes it very easy to install an EFK stack. Some people install these all at once, but I install them individually for more control over each part of the stack. We'll be creating values.yml for each part of the stack

First create the following file for Elastic Search:

Now install Elastic Search with the following commands:

Now create the following file for Fluentd:

Now install Fluentd with the following commands:

Finally create the following file for Kibana:

Now install Kibana with the following commands:

If all worked correctly, Fluentd should automagically be reading the stdout from your containers and sending them to Elastic Search. Btw, Fluentd knows where to find logs due to the default configuration found at: Fluentd Default Values.

Using Kibana On Kubernetes

At this point, you need to access your Kibana instance from a browser. I configure an ingress and service behind authentication so other devs can use Kibana, but for the sake of simplicity, let's just use port-forwarding to get to Kibana.

Now you should be able to access Kibana at http://localhost:8080

Go to the "Management" page, which is accessed by going to the gear icon. Then click on "Index Patterns" under the Kibana section. Create a new index pattern: logstash*. Then in the next step, for the "Time Filter" field, choose @timestamp. That's all it takes for Kibana to start consuming the logs.

If you're curious as to what you just did, basically, Fluentd is parsing your logs and creating indexes so you can jump to logs by index. Thoses indexes are labeled and you just told Kibana what those labels are so it can find them. Fluentd supports Elastic's Logstash pattern which is why the index pattern started with "logstash".

Now go to the "Discover" page, which is accessed by going to the compass icon. You can now look at logs by filtering. A common filter looks like:


kubernetes.namespace_name: NAME-SPACE and kubernetes.container_name: CONTAINER-NAME
      

Probably the most useful filters are the label ones: kubernetes.labels.*. It's really important that you deploy your apps with good consistent labels to make filtering easier.

Finally, to really make logging useful, you should have your developers logging everything to stdout in JSON format. Each of the JSON fields will be available as a filter. If you use consistent fields across all your apps, then you'll have an easy way to quickly find any error logs, or whatever you're looking for.

Done!

That's basically it to logging. Fairly easy to set up once you take the leap. You can save filters and create graphs of your logs if you really want. In fact, if you're logging user data, you can create some really cool graphs to represent how your users are using your apps. So the key takeaway should be that logging is really only as valuable as what you print in your logs.