How and why I built A Decentralized and Multi-Level Information System Using DJango 3.1 / Python3.8 / Postgres / Chart.js and Deployed on Ubuntu, Nginx and Gunicorn.
About me
Collins Onyemaobi is a seasoned Django/Python Developer with a keen interest in Big Data, Computer Vision, Deep Learning & Natural Language Processing. With 9 years plus IT development experience. Within projects/workplaces, I have oftentimes demonstrated strengths in several activities, this includes but not limited to excelling and recording success in architecting and managing one of the largest databases in Africa. I have also played a significant role in several other IT projects in Government, Education, and leading a team of developers in the development of an Electronic Medical Record (EMR) System used for HIV management in 94 primaries, secondary and tertiary facilities in south-eastern Nigeria.
The problem I wanted to solve
Covid-19 pandemic motivated project to automate and decentralized Data Capture Role up to 8192 locations and for multiple Organizations (18 in this case), and enforces secured login with audit trails.
The Solution - What is A Decentralized and Multi-Level Information System Portal?
A Portal for capturing, nominating, substituting, deployment, monitoring, and reporting of Agents across 8192 locations for multiple organizations (18 in this case). The system has two-level information flow and each flow is subdivided into three levels. These sublevels have multiple unique activities tied to the organization and the drill down locations.
Tech stack
I employed Django 3.1 and Python3.8 to develop the Solution because of Django's clean & rapid application development architecture. I modified the built-in Django User model to allow for email authentication. This gave me the freedom to add custom methods to the User model without having to rely on a OneToOne model or having to implement an Abstract model.
I implemented multiple user types based on the requirements of the portal as each user account is tied to an organization (18 of them) and a location(8192). To handle many user types, and users can assume multiple roles and for different locations, I opted for the option to create an extra table and created a many to many relationship:
I harnessed the django built in templating structure and queryset to transform data from a python dictionary to a safe json data for the representation of chats at different roles using chat.js.
Below is the list of Public Libraries Used:
asgiref==3.2.10 - For Python asynchronous web apps and servers to communicate with each other, and positioned as an asynchronous successor to WSGI..
Django==3.0.7 - For a high-level Python Web framework that encourages rapid development and clean, pragmatic design.
django-filter==2.3.0 - For Django-filter is a reusable Django application for allowing users to filter querysets dynamically.
django-smart-selects==1.5.6 - For Django application to handle chained model fields.
django-widget-tweaks==1.4.8 - For Tweak the form field rendering in templates, not in python-level form definitions.
Pillow==7.1.2- For Python Imaging Library (Fork)
pkg-resources==0.0.0- For A fixture lifecycle management library for your tests
psycopg2==2.8.5- For psycopg2 - Python-PostgreSQL Database Adapter
pytz==2020.1- For World timezone definitions, modern and historical
six==1.15.0- For Django Compatibility Library
sqlparse==0.3.1- For Non-validating SQL parser
The process of building A Decentralized and Multi-Level Information System Portal Using DJango 3.1 / Python3.8 / Postgres / Chart.js and Deployed on Ubuntu, Nginx and Gunicorn.
I started by gathering requirements, analyzing them. Upon concluding with the preliminary consultations, I came up with a design and timeline for the proposed work plan which was approved
At the early stages of development, I created a virtual environment, installed Django using pip3, I modified the settings.py file with the app name, database configurations, created urls.py routes. and initiated development. Throughout the development unit and integration testing was carried out.
After development, a presentation was made to the various key players. Suggestions
where noted, refinements made and final testing was carried out.
At the deployment stage, I created environment variables with the Linux infrastructure, created sockets with the Gunicorn application server to interface with our applications, and set up Nginx (performance webserver) to reverse proxy to Gunicorn, giving us access to its security and performance features to serve our apps.
At the production stage, I just monitor performance.
Challenges I faced
One major challenge was trying to modify some of the core functionality with the framework. Secondly what is the best way to approach a certain front-end task? For example, I implemented a chain select from scratch and noticed that at the fourth level of drill down the great grand child on the chain took a longer time to be populated but after a few research django-smart-selects did the magic.
Because of how complex nature of the business logic, It took me three weeks as against the two weeks earlier estimated for the development.
Key learnings
I learned a lot from this project which has to do with a handful of the enormous core functionality that already exists within the Django framework which can easily be reusable, extended, or customized.
What I would have done differently maybe to allow for the option of both email and username at login only rather than restrict printing the system to just email.
Tips and advice
While Software frameworks exist, my candid advice would be to spend time meeting different key players of the solutions to be built (Stakeholders, user department, end-users, etc) after which draft of the design, functional and project specification document is developed. Both parties should agree on deliverables and acceptance correction. This will not only assist you in choosing the right technology but also the right strategy to employ using specific technology.
And for the development user experience(UX) and business logic or functionality is the key as this would determine the level of satisfaction for the given project.
Final thoughts and next steps
As expected every project must have an end date. So the next version (2.0) would focus on the deduplication engine of the Identification of multiple person records across all organizations.
