Using Django with Elasticsearch, Logstash, and Kibana (ELK Stack)

To some developers, the concept of searching has always been to use the conventional database such as PostgresQL, MongoDB, SQLite etc and running queries on them. While this is good, there are situations where lookup has to be on a very large dataset and optimization becomes a necessity.

Chit Chat

Elasticsearch in summary is a datastore/search engine that can be used to index data from any conventional database for quick search.

With so much going on in your application and you need a way of getting all this information logged to a central location. Think Logstash, a tool to collect, process, and forward events and log messages.

When all is said and done, won't you prefer a way to see all this data and events visualized? Yes! enter Kibana.

So, What do we want to accomplish here?

• Set up the ELK stack and ensure that it works properly
• Pull down extra dependencies/libraries
• Connect the dots to an existing Django project
  • Sending Django Logs to Logstash
  • Indexing existing database to elasticsearch
  • Indexing of every new instance that a user saves to the database
  • Visualization using kibana
• A basic search example

Here is a link to a sample setup so that you can easily see what's going on. Shall we begin?

SOURCE CODE

ELK SETUP

For Ubuntu

• Install ElasticSearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html
• Install Kibana:
  https://www.elastic.co/guide/en/kibana/current/setup.html
• Install Logstash:
  https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

For Mac Using homebrew

• Install ElasticSearch: brew install elasticsearch
• Install Logstash: brew install logstash
• Install Kibana: brew install kibana

After setting up the ELK stack, to ensure that it runs properly go to default localhost:5601 and localhost:9200 for kibana and elasticsearch respectively to see that it is running.

Note: You need to start each process from the command line.

Pull down extra dependencies/libraries

For Django, we will make use of Python-logstash via pip install python-logstash a python logging handler for logstash. This will allow us send all our logs to logstash. We will also make use of django-elasticsearch-dsl module that will allow us interface with elasticsearch for this tutorial.

pip install django-elasticsearch-dsl

# Elasticsearch 5.x
pip install 'elasticsearch-dsl>=5.0,<6.0'

# Elasticsearch 2.x
pip install 'elasticsearch-dsl>=2.0,<3.0'

Connect the dots to an existing Django project

• Add the following in Django’s settings:

# settings.py 

INSTALLED_APPS = [
  # .... 
    'django_elasticsearch_dsl',
]

ELASTICSEARCH_DSL={
    'default': {
        'hosts': 'localhost:9200'
    },
}

LOGGING = {
  'version': 1,
  'disable_existing_loggers': False,
  'formatters': {
      'simple': {
            'format': 'velname)s %(message)s'
        },
  },
  'handlers': {
        'console': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
            'formatter': 'simple'
        },
        'logstash': {
            'level': 'WARNING',
            'class': 'logstash.TCPLogstashHandler',
            'host': 'localhost',
            'port': 5959, # Default value: 5959
            'version': 1, # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
            'message_type': 'django',  # 'type' field in logstash message. Default value: 'logstash'.
            'fqdn': False, # Fully qualified domain name. Default value: false.
            'tags': ['django.request'], # list of tags. Default: None.
        },
  },
  'loggers': {
        'django.request': {
            'handlers': ['logstash'],
            'level': 'WARNING',
            'propagate': True,
        },
        'django': {
            'handlers': ['console'],
            'propagate': True,
        },
    }
}

To know more about logging in Django check/read the docs here

Now that we have configured logging for our Django project we need to tell logstash where it should get the input from and how it should send that info to elasticsearch. In order to do that we need to create a logstash.conf file with the instruction for logstash there.

input {
    tcp {
    port => 5959
    codec => json
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

Note: Ensure that logstash port (5959) and elasticsearch hosts (localhost:9200) are the same as that in your settings.py

• Now run logstash using ./logstash -f path/to/logstash.conf
• Open kibana dashboard locahost:5601 and create the index using django or manually via the developer's console on Kibana's dashboard logstash-* we will use Django for this tutorial. To know how to use the console or interact with elasticsearch via the REST API, I recommend this brief video on youtube or this blog post
• Start Django server and browse a page that raises some error or make use of Django logs to see something displayed on your kibana dashboard

# import the logging library
import logging

# Get an instance of a logger
logger = logging.getLogger(__name__)

def my_view(request, arg1, arg):
    ...
    if bad_mojo:
        # Log an error message
        logger.error('Something went wrong!')

Now let's Connect ElasticSearch with Django
here is our sample models.py Book model which we intend to index using elastic search.

class Book(TimeStamp):
    """Book model defined here"""
    title = models.CharField(max_length=100)
    isbn = models.CharField(max_length=100)
    category = models.CharField(max_length=100)

    def __unicode__(self):
        return "Book Title: {}" .format(self.title)

The first thing you need to do here is to create a connection from your Django application to ElasticSearch.

create a documents.py file and add this to the file, this is where the Elasticsearch code will live.

# documents.py

from elasticsearch_dsl.connections import connections
# Create a connection to ElasticSearch
connections.create_connection()

That done we need to have a definition of what you want to be indexed into it.

# documents.py

from django_elasticsearch_dsl import DocType, Index
from .models import Book

book = Index('books') 

# reference elasticsearch doc for default settings here
book.settings(
    number_of_shards=1,
    number_of_replicas=0
)

@book.doc_type
class BookDocument(DocType):

    class Meta:
        model = Book
        fields = ['title', 'isbn', 'category']

save and run this command on your console to create the index for your application
python manage.py search_index --rebuild this will create the index for the existing model. Also, whenever a new model instance is saved the instance is automatically indexed into elasticsearch, reference the django module for more context.

Now Let's see this via Kibana
When you open localhost:5601 and navigate via the Discover tabs just select the name of your index and you'll see what has been indexed from your database, you can also perfrom custom seach here if you prefer though...

A basic search example

To create a simple search to find all books by title

add this snippet to documents.py

# documents.py 

from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search
client = Elasticsearch()
my_search = Search(using=client)

# define simple search here
# Simple search function
def search(title):
    query = my_search.query("match", title=title)
    response = query.execute()
    return response

Lets try the search feature. In the shell python manage.py shell

Python 2.7.13 (default, Dec 18 2016, 07:03:39)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from bookmeapi.documents import *
>>> print(search("Helper"))
<Response: [<Hit(books/book_document/1): {u'category': u'Fiction', u'isbn': u'233394', u'title': u'Th...}>]>
>>>

Here is the complete code for documents.py

from elasticsearch_dsl.connections import connections
from django_elasticsearch_dsl import DocType, Index
from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search

client = Elasticsearch()

my_search = Search(using=client)

from .models import Book

# Create a connection to ElasticSearch
connections.create_connection()

book = Index('books')

book.settings(
    number_of_shards=1,
    number_of_replicas=0
)


@book.doc_type
class BookDocument(DocType):

    class Meta:
        model = Book
        fields = ['title', 'isbn', 'category']


# define simple search here
# Simple search function
def search(title):
    query = my_search.query("match", title=title)
    response = query.execute()
    return response

Conclusion

We have certainly come a long way, from setting up the ELK stack to making a connection to an existing Django app to implementing a simple search. Where should we go from here? I'd suggest further read up from ElasticSearch website and a look up on Security with the ELK stack when moving from development to Production.

Thanks for reading and feel free to like this post.