InfluxDB To Grafana: Visualizing Time Series Data in Real Time

Introduction to InfluxDB:

InfluxDB is an open source distributed time series database. According to Wikipedia, it is written in Go and optimized for fast, high-availability storage and retrieval of time series data in fields such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics.

Times Series (TS) data :

TS Data is basically data related to questions that take place over time (short periods as well as long periods).
Classic examples of time series data include:

1. Stock trades and quotes over time in financial markets
2. Metrics data, example data coming from servers, or application performance data
3. User Analytics data
4. Sensor Data, example: measurements coming from physical sensors

Why do we need a time series database?

Some developers may debate that instead of having a time series database, we could just use a MYSQL database with a time column and create an index by that.

Here are some of the reasons why we should use time series database, such as InfluxDB, instead of a MYSQL database :

1. Scale: Let’s try to understand it through an example of DevOps. Let's say we have 2000 servers, VMs, Containers or sensor units, each performing 200 measurements per server. Our task is to sample them once every 10 seconds. This gives us a total of 3,456,000 distinct points (equal to approx. 3.5 Billion data points) per day. It would be close to impossible to put it all in a MySQL table.

2. Sharding Data: Usually, time series data is done at scale, which means we have to build it in a distributed system. Even if you're using a database like Cassandra, you would still have to figure out how to shard your data. You will most likely end up writing application level code to do that.

3. Data Retention: It’s very common in time series to have high precision data that is kept around for a short period of time and lower precision data that is kept for months or years. Usually, if you are using some other database, you would have to write some application level code to do that.

4. Rollups and aggregation: If you have high precision data every 10 seconds, you would want to roll it up into lower precisions (say 10 minutes or hourly views) that you can keep around for a long time. Again, you'd usually have to write the code to do that.

InfluxDB has a number of features that can take care of all the features mentioned above automatically.

Features of InfluxDB:

1. SQL style query language: This makes it easier to use and to work with.

2. Retention Policies: InfluxDB uses retention policies to handle your data retention periods automatically. You can designate an area of database to keep data for 7 days and have another area keep data for 6 months, 2 years, 5 years, etc.

3. Continuous Queries: This feature lets you make any query run as a continuous query, which essentially means telling InfluxDB to run the query in the background and compute it automatically. That way, we can do rollups and aggregation using this feature.

4. HTTP API - 2 endpoints: InfluxDB gives two simple endpoints, where we specify the database name and the retention policy, with which you are writing or reading in data.

Write EndPoint Syntax:

	 HTTP POST	/write?db=mydb&rp=foo

Read EndPoint Syntax:

	 HTTP GET 	/query?db=mydb&rp=foo&q=querystring

Introduction to Grafana:

Grafana is a data visualization tool that provides ways to create, explore, and share data in easy to understand graphical representation. It's mainly used to visualize time series data. It supports Graphite, ElasticSearch, Prometheus, InfluxDB, OpenTSDB, and KairosDB.

Let us now build a demo, where we have live streaming data coming in from a torrent client. This data will be inserted into influxDB at a regular time interval and will be analyzed in real time and visualized inside Grafana dashboard.

Steps to Visualize Time Series Data in Grafana from InfluxDB:

1. Download a sample torrent file. I took the sample torrent file of Ubuntu from the Ubuntu downloads page: Ubuntu download link

2. Run Deluge (download from http://deluge-torrent.org/), a torrent client.
   Then we need to make deluge accessible via API , so click preferences,

Then, select plugin from left side bar, and enable WebUI.

Then, select "WebUi" from the left sidebar and check enable web interface at port 8112.

This will enable us to use Deluge in our browser at http://localhost:8112.

Next, add the torrent of Ubuntu we downloaded in step 1 to Deluge.

3. Once we have added the torrent file, Deluge will start downloading the file.

4. We need to control the speed of download in order to observe the data in Grafana later. We will change the download speed to minimum (i.e. 5KB / s)

5. Open another terminal window and let's start InfluxDB. The command to run influxdb:

    influxd -config /usr/local/etc/influxdb.conf

Next, we'll create a database in InfluxDB with the name, ‘deluge’. Here is the curl command for the same:

create database deluge

Another way to run commands in InfluxDB is by calling its REST api using curl:

curl -i -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE deluge"  

6. Next, we will clone this repository: https://github.com/mvantassel/deluge2influx

There needs to be a quick change in the file deluge2influx.js. The function writeToInflux needs to be written like so:

function writeToInflux(seriesName, values, tags, callback) {
    console.log("seriesName = ", seriesName);
    console.log("values = ", values);
    console.log("tags = ", tags);

    return influxClient.writePoints([{
            measurement: seriesName,
            fields: values,
            tags: tags
        }], callback)
}

The same database name, along with the username and password to login to InfluxDB, has to be specified in the file deluge2influx.js:

const influxClient = new Influx.InfluxDB({
    host: 'localhost',
    port: '8086',
    protocol: 'http',
    username: 'root',
    password: 'root',
    database: 'deluge'
})

Additionally, we need to specify the setting to connect to Deluge client in the following code in the same file:

const delugeConfig = {
    host: process.env.DELUGE_HOST || 'localhost',
    protocol: process.env.DELUGE_PROTOCOL || 'http',
    port: process.env.DELUGE_PORT || 8112,
    password: process.env.DELUGE_PASSWORD || 'deluge'
};

Once we have made the update, run the app using:

node deluge2influx.js

Here is the screenshot of the terminal once the app began running.

7. What this app does is that it reads the data while it's being downloaded by Deluge torrent client. It then dumps the data an InfluxDB specified database. We can see from the screenshot below that regular POST commands are being called to dump data into InfluxDB:

8. Once the app began executing, we can go and check in InfluxDB whether data is actually coming into the database from deluge.

These screenshots show that data is ogoing into the Deluge database and getting stored in the tables named new_torrent and new_torrents.

Setup Grafana:

Now, let us move onto visualise the data that is being dumped into InfluxDB at regular intervals. Download Grafana and set it up as mentioned on its website.

To run grafana as a service, run this command:

brew services start grafana

You should have Grafana up and running at http://localhost:3000/login

1. We need to create a new dashboard and add a new user to it.

2. Install InfluxDB plugin in Grafana.

3. Add a SQL query to read data from InfluxDB at regular intervals. See format of query below:

4. Update the datasource details to match the InfluxDB database name as shown in the screenshot below:

Enter the login credentials for InfluxDB in dashboard.

5. We then need to specify the query in the metric tab of dashboard.

Choose ‘distinct’ as the aggregation parameter for field ‘downloaded’

6. Also specify the data-source created earlier.

You may want to check the settings of data-source just to be sure it is the same we entered in the dashboard.

7. Once everything is setup properly, we should see the visualization appear in the dashboard as shown in the following screenshot.

These are some of the screenshots of how visualized data looks like. It keeps pulling data out of InfluxDB at regular interval of 1 minute, as specified, while building the query in dashboard.

That's it! I hope this tutorial helps you get a better understanding of InfluxDB, Grafana and Time Series data.