Offline With SlashDB

Ever wish you can download complex, parent-child relationship (hierarchical) data from a relational data store and persist that data locally for use from browser or mobile device session to session? In a prior post, we looked at a simple key/value method for bringing simple SlashDB-sourced scalar (attribute/value) data into a local storage area in your browser using the Web Storage concept for a client-side alternative to using cookies. In this article, we’ll show you how to use an in-browser database called IndexedDB to harness the data nesting powers of SlashDB and create a “kid-sister” version of your relational database within a powerful database using a concept of object or document databases.

A new thinking is getting a rich following today in an area that is called “Offline First” (see www.offlinefirst.org). For users of SlashDB, where applications rely on SlashDB’s ability to turn a database into an online resource via API access, we’d like to have our apps always available, even if networks aren’t. Today’s application users expect a seamless, responsive experience no matter where they are. However, traditional applications (web and mobile) can get bogged down or stop working altogether if networks are congested, slow, or interrupted. In an “offline first” approach, data is handled on the device / client browser and synced to the server when networks become available.

SlashDB Meets IndexedDB

Because we often like to work with data formatted as objects, specifically as JSON objects, IndexedDB is the perfect partner to SlashDB’s output structure in that the data is client-stored as Object Stores.   Plus, with IndexedDB, all requests to SlashDB are executed within the context of a “transaction”.   This is important because if you create a transaction for a series of steps and one of the steps fails, none of the actions are applied. Because the IndexedDB databases are Object Stores, you can create objects for each kind of “thing” you want to store.f These Object Stores can have multiple values such as strings, dates, arrays or Javascript objects. The Object Store can be built with specific fields that are indexed (as we’ll see below), sorted, or even filtered by choice fields and values.

The raw IndexedDB API is a bit tricky to work with. This example will include this API but it’s important to mention here that Jake Archibald of Google has created an “IndexedDB Promised” library which contains wrappers around the IndexedDB API that uses Javascript Promises (more at: https://developers.google.com/web/fundamentals/primers/promises ).

IndexedDB is an older, asynchronous API that uses callbacks and fires event to signal completions and errors.

Looking at the SlashDB “depth” Parameter

SlashDB has an additional parameter: depth = [n] .

Using this parameter returns resources with all related resources. The number (n) indicates how many subsequent relations must be included (nested) in result.

For example, given this partial database schema for the Chinook database:

… we can issue:

http://demo.slashdb.com/db/Chinook/Artist/ArtistId/51.json?depth=1

… to return a JSON object with all the Queen albums – data from the Artist and Album base tables.

Note: our demo SlashDB instance has implemented an API for a sample Chinook database model.  Details of this relational database example project can be found at https://github.com/lerocha/chinook-database . The Chinook data model represents a digital media store (e.g. iTunes), including tables that include artists, their albums and a plethora of album track data.

Moving down the hierarchy to the Track table is as simply as issuing the above API call with a depth=2 parameter.  Data is now available for the Artist, Album, and Track base tables.

Extracting from SlashDB into the Object Store

There exists very detailed introductory IndexedDB material and technical specifications on the Mozilla web site at:

https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Basic_Concepts_Behind_IndexedDB

If you’ve used NoSQL databases before, IndexDB will be familiar to you.

Below, we’ll show you a working example that includes the creation of a database and a data store (it’s like a relational table) of all track information on a per album basis for the 70’s rock band Queen. In addition, you can add indexes of the data. An index lets you look up the values stored in an object store using the value of a property of the stored object, rather than the object’s key. The format of the index creation code is:

your_objectStore_name.createIndex("column_name", "column_name", { unique: false });

…and we’ll see an example in the code sample below.

Worthy of another article is a concept called the Service Worker. With Service Workers, your users no longer need to face the dreaded “You Are Not Connected to the Internet” screen anymore. Using Service Workers, Javascript worker scripts that run in the background, you can intercept and cache any network requests to and from your site. Whether you are building websites for mobile, desktop or tablet devices, you have control over how you want to respond to requests – with or without a network connection.   The basic architecture of the offline first concept can be portrayed as:

And the code that creates and populates the object store from SlashDB-sourced data is:

<!DOCTYPE html>
<html>

<head>
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script>
    <link rel="stylesheet" href="https://ajax.googleapis.com/ajax/libs/jqueryui/1.11.4/themes/smoothness/jquery-ui.css">
    <script src="https://ajax.googleapis.com/ajax/libs/jqueryui/1.11.4/jquery-ui.min.js"></script>
</head>

<body>
    <h1 class="page-start">Populate the IndexedDB database from SlashDB server-side data</h1>
    <form role="form" onsubmit="addTracks()">
        <button type="submit" class="btn btn-default">Add tracks</button>
    </form>

    <script type="text/javascript">
        // When the document is ready execute the JQuery and open an IndexedDB database and create the object store
        // when there is no upgrade needed;  add an index on the Composer field
        $("document").ready(function() {



            if (!window.indexedDB) {
                window.alert("Your browser doesn't support a stable version of IndexedDB. Such and such feature will not be available.");
            }

            // Open the database
            var request = window.indexedDB.open("trackData", 1);

            // error opening database
            request.onerror = function(event) {
                alert("Why didn't you allow my web app to use IndexedDB?!");
            };

            // success in opening database
            request.onsuccess = function(event) {
                db = event.target.result;
                console.log('successfully opened the Database...')
            };

            // Creating the object store called "tracks" with a keyPath named TrackId
            request.onupgradeneeded = function(event) {
                var db = event.target.result;
                console.log('creating object store...');

                var objectStore = db.createObjectStore("tracks", {
                    keyPath: "TrackId"
                });
                // Create an index to search the tracks by the composer name (left as an exercise for you)
                // we may have duplicates so we can't use a unique index.
                objectStore.createIndex("Composer", "Composer", {
                    unique: false
                });
            };



        });
        //  Add the records into the object store
        function addTracks() {
            console.log('adding tracks in this function')
            // but first let's get the track information via SlashDB API call via XHR
            var allTracksForArtist = getTrackInfofromSlashDB();
            // Store values in the  objectStore.
            var tracksObjectStore = db.transaction("tracks", "readwrite").objectStore("tracks");
            allTracksForArtist.forEach(function(artistsTrackData) {
                tracksObjectStore.add(artistsTrackData);
            });


        }; // end function addTracks

        function getTrackInfofromSlashDB() {
            var allTracksForArtist = [];

            var SlashDBrequest = new XMLHttpRequest();
            // note that the "async" boolean parameter of the open method is false in order to
            // NOT perform the operation asynchrononously
            SlashDBrequest.open("GET", "http://demo.slashdb.com/db/Chinook/Artist/ArtistId/51.json?depth=2", false);
            SlashDBrequest.setRequestHeader("Content-Type", "text/xml");
            SlashDBrequest.addEventListener("load", function() {
                console.log(SlashDBrequest.response);
            }, false);

            try { // when 'offline' don't run the send method
                SlashDBrequest.send();
            } catch (error) {
                // do nothing
            }


            if (SlashDBrequest.response !== "") {
                var obj = JSON.parse(SlashDBrequest.response);

                console.log(obj);

                var numberOfAlbums = obj.Album.length;

                for (var albumIx = 0; albumIx < numberOfAlbums; albumIx++) {
                    numberOfTracks = obj.Album[albumIx].Track.length;
                    for (var trackIx = 0; trackIx < numberOfTracks; trackIx++) {
                        allTracksForArtist.push({
                            TrackId: obj.Album[albumIx].Track[trackIx].TrackId,
                            Composer: obj.Album[albumIx].Track[trackIx].Composer,
                            Bytes: obj.Album[albumIx].Track[trackIx].Bytes,
                            Milliseconds: obj.Album[albumIx].Track[trackIx].Milliseconds,
                            GenreId: obj.Album[albumIx].Track[trackIx].GenreId,
                            Name: obj.Album[albumIx].Track[trackIx].Name,
                            MediaType: obj.Album[albumIx].Track[trackIx].MediaType
                        });
                    } // for loop for tracks

                } // for loop for albums by the artist


            } // if


            return allTracksForArtist;
        }
    </script>
</body>

</html>

And now lets see the results of the actions above within the IndexedDB database and Object Store.  By launching the Chrome DevTools we can navigate to the Storage section of the Application item and view the IndexedDB databases and associated Object Stores and Indexes:

Inspecting the indexed value (Composer) reveals:

What Can We Do with Network Failures and Delays with SlashDB and an Offline First Mindset

How can we best deal with latency in the network or actual network failures? If there is a network failure, we can get fall back content in the form of cached results – the last data we could have from the network.   Unfortunately in this approach we have to actually wait for that network failure before we start to show content. The alternative is to attempt to get as much cached data on the user’s screen as possible using the IndexedDB data. We will go to the network, sure, but we’re not going to wait for the network. The idea is to update the page first, with that cached data, and then get content later, if available from the network. When the fresh data is received from the network, then we can refresh what the user is currently looking at as well as save the newly received remote data into the IndexedDB cache again.

The less the user has to worry about connectivity, the better. SlashDB and IndexedDB – putting assets of data closer to the user.