Timelion in the ELK Stack
I am sure that you are pretty familiar with “ELK Stack”, but, if you are like me, “Timelion” wasn’t one the first things that you heard about it. What’s is it? and Why should I have some idea about it?
According the official documentation, Timelion is a tool for visualization of temporal series in kibana, but with a different approach, using a specific syntax and being able to chain functions … but, to better understand what we are talking about, let see it in action with some queries and examples. So, let’s start on it!
Timelion #00: Defining one Index
Timelion has their own syntax enable drawing data from different indices or data sources into one graph. The way to define on what index we are going to work and make queries is the following:
Timelion #01: Sample on a “Twitter” Dataset
Let’s do something simple, to see a little more. Let’s say that we want to work on a dataset with just six records, and the structure de each record is very straightforward: user, age, post-date and message… and nothing else
And we could add that in elastic with some post in “Dev Tools”:
For the sample, I would use these as “my-twiteer” dataset, they are only six simple records with messages popular super-heroes:
"user" : "thor",
"edad" : 32,
"post_date" : "2018-01-31T20:12:12",
"message" : "Prueba en Elasticsearch Tweet1 (Thor)"
"user" : "rocky",
"edad" : 48,
"post_date" : "2018-01-31T20:13:12",
"message" : "Prueba en Elasticsearch Tweet1 (Rocky)"
"user" : "Meteoro",
"edad" : 16,
"post_date" : "2018-01-31T20:14:12",
"message" : "Prueba en Elasticsearch Tweet1 (Meteoro)"
"user" : "Batman",
"edad" : 29,
"post_date" : "2018-02-03T20:12:12",
"message" : "Prueba en Elasticsearch Tweet4 (Batman)"
"user" : "Aquaman",
"edad" : 31,
"post_date" : "2018-02-01T20:13:12",
"message" : "Prueba en Elasticsearch Tweet5 (Aquaman)"
"user" : "Green Lanter",
"edad" : 42,
"post_date" : "2018-02-08T20:14:12",
"message" : "Prueba en Elasticsearch Tweet6 (Green Lanter)"
And, after submit this “posts”, we should be able to query this records with the GET method:
Let’s see a first insight: What is the “Average edad” of this heroes? Let’s see it inside “visualize” section to validate that they are already in elasticsearch:
Now, let’s move to Timelion.
First, we need to find “my-tweeter” data, so I have to do a first time configuration by my own::
As we can read, we should do for first time:
“To search other indices, go to Management / Kibana / Advanced Settings and configure the
timelion:es.timefieldsettings to match your indices.”
So, I will change it:
And the result is OK! I have my data indexed on Timelion:
but, in deed, It is not necessary to edit the advanced settings… first: I will restore the “Timelion Advances Settings” to previous and default values, and let’s continue with our own index:
…we can add index and timefield on .es() function
What is the average “edad” (age) in my-tweets dataset?
.es(index=my-twitter, timefield=post_date, metric=avg:edad)
.points(radius=12, fill=1, fillColor=#009900)
So, with this simple case, we have a first approach what is Timelion, and how to work on in.
Let’s continue with a second dataset, a little more complex:
Timelion #02: Sample over a “AIR Quality” Dataset
Let’s change the dataset to a public dataset on AIR Quality on Buenos Aire. We are interested on the public and raw data, and it is available on https://data.buenosaires.gob.ar/dataset/calidad-de-aire:
If we want to see some visualization available on this dataset, we have a first look on the official Open Data page:
If we are intereseted on more complex visualization on this dataset, we have also some relate web site:
Centenario, Buenos Aires, Argentina Air Pollution: Real-time PM2.5 Air Quality Index (AQI)
How polluted is the air today? Check out the real-time air pollution map, for more than 60 countries.
And the following links: https://aqicn.org/map/buenos-aires/es/#@g/-34.6288/-58.4469/12z
But, we are interested on what we could make with Timelion, so, let’s see some queries on Timelion, with two index on the same graph:
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_CO),
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_NO2),
It could be nice to have some kind of “normalization” for High values, another Idea is compare same variable on different months, and to have two different indexes ….so, we will add “moving average” and two axis:
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_CO).mvavg(7).yaxis(1),
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_NO2).mvavg(7).yaxis(2),
Also, we could add use a “label”, and it is done with another function concatenated:
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_CO).mvavg(7).label("AIRQ_CO"),
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_NO2).mvavg(7).yaxis(2).label("AIRQ_NO2"),
label() + trend()
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_CO).mvavg(7).label("AIRQ_CO").trend(),
.es(index=uela-dataset-01-all, timefield=FECHA_HORA, metric=max:AIRQ_NO2).mvavg(7).yaxis(2).label("AIRQ_NO2").trend(),
Legend in a row with:
.... .trend().legend(columns=4, position=nw),
Legend with “if”, it is only valid the structure for legibility purposes:
Ok, with this dataset, we could make some queries in Timelion syntax language, and we could see in action how this works.
Timelion #03: Some Tipical Queries
Now, that we are familiar with basic Timelion queries, let’s add another queries a little more complex, this queries are well explained in the official video, but I am only bring to the post the query on the datase
And you can see the moment (in minutes) inside the video where the query is explained
And if we add this queries to our dataset, we can share may pictures with the Timelion expression and the associated visualization graph:
(Monday and Monday should be an example of Month and Month, whit our banking transactions we can compare how we spent our money in different months.)
Timelion #04: Filling Gaps
Let’s assume that you need to filling the gaps, so, in Timelion, the following pictures show how to deal with it:
Itervals (timelion fills …)
Timelion #05: Moving Average
What about if we ban to obtain trend lines, like the moving average or another one?
Let use the las functions: if(), gt() and points():
Ok! I will stop the post at this point. We have been working with two dataset, the first one very simple, the second one just a little more complex.
Later, we make some typical queries and we follow some advices taken and samples taken from the official docs
I hope you have enjoyed learning something new, and I expect that the samples helped you to see how is the way to work with it.
See you in another one!
These are some nice videos and resources that I had been reading when I wrote this post:
- The video: “Timelion: Magic, Math, and Everything in the Middle”
- The excellent “Tutorial de Timelion: De cero a héroe” by Tim Roes
- The official “Timelion Getting Started”
- And a Public Dataset about Co2 emission with their own dashboard