Timelion in the ELK Stack

7 min readFeb 12, 2018

I am sure that you are pretty familiar with “ELK Stack”, but, if you are like me, “Timelion” wasn’t one the first things that you heard about it. What’s is it? and Why should I have some idea about it?

According the official documentation, Timelion is a tool for visualization of temporal series in kibana, but with a different approach, using a specific syntax and being able to chain functions … but, to better understand what we are talking about, let see it in action with some queries and examples. So, let’s start on it!

Timelion #00: Defining one Index

Timelion has their own syntax enable drawing data from different indices or data sources into one graph. The way to define on what index we are going to work and make queries is the following:

.es(index=uela-dataset-01-geo-all)

Timelion #01: Sample on a “Twitter” Dataset

Let’s do something simple, to see a little more. Let’s say that we want to work on a dataset with just six records, and the structure de each record is very straightforward: user, age, post-date and message… and nothing else

And we could add that in elastic with some post in “Dev Tools”:

For the sample, I would use these as “my-twiteer” dataset, they are only six simple records with messages popular super-heroes:

PUT /my-twitter/tweet/1?pretty
{
    "user" : "thor",
    "edad" : 32,
    "post_date" : "2018-01-31T20:12:12",
    "message" : "Prueba en Elasticsearch Tweet1 (Thor)"
}PUT /my-twitter/tweet/2?pretty
{
    "user" : "rocky",
    "edad" : 48,
    "post_date" : "2018-01-31T20:13:12",
    "message" : "Prueba en Elasticsearch Tweet1 (Rocky)"
}PUT /my-twitter/tweet/3?pretty
{
    "user" : "Meteoro",    
    "edad" : 16,
    "post_date" : "2018-01-31T20:14:12",
    "message" : "Prueba en Elasticsearch Tweet1 (Meteoro)"
}PUT /my-twitter/tweet/4?pretty
{
    "user" : "Batman",
        "edad" : 29,
    "post_date" : "2018-02-03T20:12:12",
    "message" : "Prueba en Elasticsearch Tweet4 (Batman)"
}PUT /my-twitter/tweet/5?pretty
{
    "user" : "Aquaman",
            "edad" : 31,
    "post_date" : "2018-02-01T20:13:12",
    "message" : "Prueba en Elasticsearch Tweet5 (Aquaman)"
}PUT /my-twitter/tweet/6?pretty
{
    "user" : "Green Lanter",
            "edad" : 42,
    "post_date" : "2018-02-08T20:14:12",
    "message" : "Prueba en Elasticsearch Tweet6 (Green Lanter)"
}

And, after submit this “posts”, we should be able to query this records with the GET method:

Let’s see a first insight: What is the “Average edad” of this heroes? Let’s see it inside “visualize” section to validate that they are already in elasticsearch:

Now, let’s move to Timelion.

First, we need to find “my-tweeter” data, so I have to do a first time configuration by my own::

As we can read, we should do for first time:

“To search other indices, go to Management / Kibana / Advanced Settings and configure the timelion:es.default_index and timelion:es.timefield settings to match your indices.”

So, I will change it:

And the result is OK! I have my data indexed on Timelion:

but, in deed, It is not necessary to edit the advanced settings… first: I will restore the “Timelion Advances Settings” to previous and default values, and let’s continue with our own index:

…we can add index and timefield on .es() function

.es(index=my-twitter, timefield=post_date).color(#ee1122)

.es(index=my-twitter, timefield=post_date).color(#ee1122)

What is the average “edad” (age) in my-tweets dataset?

.es(index=my-twitter, timefield=post_date, metric=avg:edad)
     .color(#ff0000)
     .points(radius=12, fill=1, fillColor=#009900)

.es(index=my-twitter, timefield=post_date, metric=avg:edad)
.color(#ff0000)
.points(radius=12, fill=1, fillColor=#009900)

So, with this simple case, we have a first approach what is Timelion, and how to work on in.

Let’s continue with a second dataset, a little more complex:

Timelion #02: Sample over a “AIR Quality” Dataset

Let’s change the dataset to a public dataset on AIR Quality on Buenos Aire. We are interested on the public and raw data, and it is available on https://data.buenosaires.gob.ar/dataset/calidad-de-aire:

If we want to see some visualization available on this dataset, we have a first look on the official Open Data page:

If we are intereseted on more complex visualization on this dataset, we have also some relate web site:

Centenario, Buenos Aires, Argentina Air Pollution: Real-time PM2.5 Air Quality Index (AQI)

How polluted is the air today? Check out the real-time air pollution map, for more than 60 countries.

aqicn.org

And the following links: https://aqicn.org/map/buenos-aires/es/#@g/-34.6288/-58.4469/12z

But, we are interested on what we could make with Timelion, so, let’s see some queries on Timelion, with two index on the same graph:

.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_CO), 
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_NO2),

.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_CO),
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_NO2),

It could be nice to have some kind of “normalization” for High values, another Idea is compare same variable on different months, and to have two different indexes ….so, we will add “moving average” and two axis:

.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_CO).mvavg(7).yaxis(1),
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_NO2).mvavg(7).yaxis(2),

.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_CO).mvavg(7).yaxis(1),
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_NO2).mvavg(7).yaxis(2),

Also, we could add use a “label”, and it is done with another function concatenated:

.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_CO).mvavg(7).label("AIRQ_CO"),
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_NO2).mvavg(7).yaxis(2).label("AIRQ_NO2"),

label() + trend()

.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_CO).mvavg(7).label("AIRQ_CO").trend(),
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_NO2).mvavg(7).yaxis(2).label("AIRQ_NO2").trend(),

Legend in a row with:

.... .trend().legend(columns=4, position=nw),

Legend with “if”, it is only valid the structure for legibility purposes:

Ok, with this dataset, we could make some queries in Timelion syntax language, and we could see in action how this works.

Timelion #03: Some Tipical Queries

Now, that we are familiar with basic Timelion queries, let’s add another queries a little more complex, this queries are well explained in the official video, but I am only bring to the post the query on the datase

And you can see the moment (in minutes) inside the video where the query is explained

And if we add this queries to our dataset, we can share may pictures with the Timelion expression and the associated visualization graph:

#sample 4~ min 21:23 (Diff Monday & Monday)

(Monday and Monday should be an example of Month and Month, whit our banking transactions we can compare how we spent our money in different months.)