Firebase Database Modeling: Asset Tracker case study (part 1)

Introduction

Every modern application has a need or two for data persistence, it's never good UX for users to come back to your app only to find out all their data is lost. That's a quick way to get an app uninstalled. This basic requirement can be satisfied by persisting user-created objects in a database such as SQLite or any ORM of choice in Android. Other forms of user-generated data can be persisted using the appropriate storage option. Data is saved in Tables in the database where each table is made up of roles and columns. A table is created using a schema, schemas define how data is modeled in the database. We won't go into data modeling for SQL databases in this post but look up this page to learn more about SQLite database in Android.

It suffices to say that the list of assets in the database can be pulled with a query such as:

SELECT * FROM assets;

More so, in a fast-paced world, a mobile application is required to offer services to users in real-time. Tracker Apps like Uber update the client app every second or even less with new locations for each vehicle, this surely requires more than a single query and attempting to make such requests every second will drain users battery and bandwidth, cause the app to consume much system resource or even crash which are all undesirable. Hence the need for a real-time database.

The firebase real-time database is a NoSQL real-time database. Real-time in that data is pushed to the client app automatically when there is a change. This is made possible as the client app observes the database for changes rather than querying it. However, this does not eliminate the need for queries in complex database operations such as sorting data which are common in SQL databases. Data in the database is saved as JSON trees and when a new child is added it becomes a node in the existing JSON structure with an associated key. You can provide your own keys, such as user IDs or semantic names, or they can be provided for you using push().

An example data in the database goes like:

{
  "assets": {
    "pmt1": {
      "name": "pearl transit 01",
      "lat": 6.5244,
      "long": 3.3792,
      "passengers": { 
      	  "jane": {
          	  "name": "Jane Doe",
              "age": 24
          }
          "john": {
          	  "name": "John Doe",
              "age": 24
          }
          ...
      }
    },
    "pmt2": { ... },
    "pmt3": { ... }
  }
}

Obtaining real-time updates of data changes is as simple as creating a database reference to the location you want to listen for changes and attaching an ValueEventListener. Like this:

 DatabaseReference mAssetReference;
 mAssetReference = 
 FirebaseDatabase.getInstance().getReference().child("assets");
 ValueEventListener assetListener = new ValueEventListener() {
     @Override
     public void onDataChange(DataSnapshot dataSnapshot) {
         List<Asset> assets = new Arraylist<>();
         // obtain list of java objects from the datasnapshot
         for (DataSnapshot postSnapshot: dataSnapshot.getChildren()) {
            Asset asset = dataSnapshot.getValue(Asset.class);
            assets.add(asset);
        }
     }
 };
 mAssetReference.addValueEventListener(assetListener);

Problem

The JSON model above works okay at first when the number of assets is small, but as the app begins to scale, this model becomes very inefficient due to the way data is loaded from the firebase real-time database. Firebase Real-time Database allows nesting data up to 32 levels deep, which makes it seem like this should be the default structure. However, when you fetch data at a location in your database, you also retrieve all of its child nodes.

To retrieve an event, we specify a path to the data we want. In the example above, we specified a path to the root of the database which is a bad practice (more on this later)

 FirebaseDatabase.getInstance().getReference().child("assets");

If we wanted to retrieve a specified asset then the path will look like this:

 FirebaseDatabase.getInstance().getReference().child("assets").child(pmt1);

then we attached a ValueEventListener to this path. As a result whenever there a change anywhere in the database, a snapshot of the whole database is downloaded by the client app wasting users bandwidth.
Furthermore, we received a list of assets to display to the user, all we need here is probably only the name and location details of the asset but we ended up downloading the whole object including the nested passengers object and the users object nested inside the passengers object which we don't need for our list. We don't want to waste our users time to load unnecessary data when they only just wanted to browse a list of assets now do we?. This shows that this is a really bad model for our database.
Also, there two types of event listeners:

• ValueEventListener
• ChildEventListener

The ValueEventListener is useful when you need to process the entire list upon modification of a child as it always returns a list. When there is a single data at a location, it returns a list containing only one data. ValueEventListener has two methods:

• addListenerForSingleValueEvent(): The onDataChange method is called once when the listener is attached, then it's discarded.

• addValueEventListener(): The onDataChange method is triggered once when the listener is attached and again every time the data, including children, changes. It is not discarded and will continue listening for changes until it's manually discarded with the removeEventListener method.

Child events are triggered in response to specific operations that happen to the children of a node from an operation such as a new child added through the push() method or a child being updated through the updateChildren() method. Each of these together can be useful for listening to changes to a specific node in a database.
ChildEventListener is optimized for listening to lists of data, it is called once for each child on the list when the listener is attached and again each time a child is added or removed from the list. it always returns a single object.

Solution

A great way to resolve this situation is to break up our model and create collections at the root of the database. this is known Data fan out. Also we use a ChildEventListener instead for the list of assets.

{
  "assets": {
    "pmt1": {
      "name": "pearl transit 01",
      "lat": 6.5244,
      "long": 3.3792,
    },
    "pmt2": { ... },
    "pmt3": { ... }
  }
  
  "passengers": { 
  	 "pmt1": {
      	 "jane": true
         "john": true
      }
      "pmt2": { ... },
      "pmt3": { ... }
   }
   
   "users": {
   	  "jane": {
         "name": "Jane Doe",
         "age": 24
      }
      "john": {
         "name": "John Doe",
         "age": 24
      }
   }   
}

Notice that we now have a flat data model instead of nesting which allows us to fetch a list of assets that only provides us with the name and location details:

 DatabaseReference mAssetReference;
  mAssetReference = 
  FirebaseDatabase.getInstance().getReference().child("assets");
  ValueEventListener assetListener = new ValueEventListener() {
      @Override
      public void onDataChange(DataSnapshot dataSnapshot) {
          List<Asset> assets = new Arraylist<>();
          // obtain list of java objects from the datasnapshot
          for (DataSnapshot postSnapshot: dataSnapshot.getChildren()) {
             Asset asset = dataSnapshot.getValue(Asset.class);
             assets.add(asset);
         }
      }
  };
  mAssetReference.addValueEventListener(assetListener);

and we can get the list of passengers of a specified asset as follows:

 DatabaseReference mPassengerReference;
  mPassengerReference = 
  FirebaseDatabase.getInstance().getReference().child("passengers/pmt1");
  ChildEventListener passengerListener = new ChildEventListener() {
     @Override
     public void onChildAdded(DataSnapshot dataSnapshot,
     				String previousChildName){
        // it called for both users on the list
        // "jane": true
        // "john": true
     }
     ...
     ...
     ...
     ...
 };
 mPassengerReference.addChildEventListener(passengerListener);

This however only returns a reference to user, a complete passenger details can be retrieved by adding a singleValueEventListener inside the onChildAdded method of the PassengerReference as follows:

 DatabaseReference mPassengerReference;
 DatabaseReference mUserdetailReference;
  mPassengerReference = 
  FirebaseDatabase.getInstance().getReference().child("assets/pmt1");
  ChildEventListener passengerListener = new ChildEventListener() {
     @Override
     public void onChildAdded(DataSnapshot dataSnapshot,
     				String previousChildName){
         // it called for both users on the list
         mPassengerReference = FirebaseDatabase.getInstance().getReference().child("assets/" 
        + datasnapshot.key);
         mPassengerReference.addListenerForSingleValueEvent(new  ValueEventListener() {
              @Override
               public void onDataChange(DataSnapshot dataSnapshot) {
                    // Obtain user data
               }
        });
    }
    ...
    ...
    ...
    ...
 };
 mPassengerReference.addChildEventListener(passengerListener);

Data fan out this way ensures only the needed data is requested thereby optimizing load time.

Denormalization: How, When and Why?.(part 2)

denormalization is the duplication of data to optimize read performance
check it out here