The State of Offline In Web Applications (2017)
Application Cache
Yes, we all know it. AppCache is a douche bag! But sadly if you want your offline application to load under Safari, IE11, or Edge, it's your only option.
If you check out Mozillas documentation on AppCache, you will find a big red disclaimer saying keep away!
Using the application caching feature described here is at this point highly discouraged; it’s in the process of being removed from the Web platform. Use Service Workers instead.
But sadly take a look at can I use it for service workers. You want your offline application to work on IPad, IE, or Edge? Sorry buddy, no dice!
So, how does Application Cache work?
The afore mentioned posts lay this out in detail, but here are the basics:
- include a manifest attribute on the HTML tag of any page you wish to cache
<html manifest="/myapp/application.manifest">
If your application is a Single Page Application (SPA), then congratulations you've enabled caching on all pages, otherwise you'll need to put this on each page you wish to cache. Specifying the manifest only causes the served page to be cached.
- You'll need to place anything it references in the
application.manifestfile.
CACHE MANIFEST
CACHE:
/build/app.js
/build/app.css
/images/banner.png
NETWORK:
*
# Updated 2017-08-26T20:26:30.181Z
- There are more than a few gotchas to keep in mind with app cache
-
You can only cache resources stored under the same origin; so, if you were planning to use a CDN to serve up libraries and media files, better hope you don't need those offline.
-
You must use https and your certs better check out. Using https is something you should do anyways, but one really annoying thing (at least under Chrome) is that the certs must be signed by a trusted CA and must be associated with the domain used to access the site. Simply telling chrome to proceed with the unsafe connection is not enough. This makes development a pain. I've dealt with this is by registering local.[mydomain].com with my domain registrar, pointing the entry to 127.0.0.1 and including local.[mydomain].com in my cert.
- This kind of sucks. Allowing public DNS lookups to resolve to localhost or anything on the local network is not great. I watched a pretty scary talk on this at defcon 25 called There's no place like 127.0.0.1 - Achieving reliable DNS rebinding in modern browsers
-
App Cache does not allow much control and labors under the assumption that you want to show your content as quickly as possible regardless of how outdated or wrong it is; so, when a page that was previously cached is hit, the browser loads immediately from app cache regardless of your connection status. Only after the page loads does it start checking to see if the manifest has changed. If it decides it has then it will start checking the files in the manifest for changes and asynchronously downloading them in the background. When done an event is fired. Only at that point will reloading the page show the user an up to date page.
The best way I've found to deal with this is to listen for this event and prompt the user to reload the page to get a new version. This process is flaky at best. Sometimes the event fires almost immediately other times, it takes five minutes to never. This means your backend better be ready to handle requests delivered from old client code, that's fun!
-
Be careful about the headers that you send with the application.manifest file. Application Cache will not re-check the files included in your manifest for changes unless it detects that the manifest file its self has changed! In my sample manifest you may have noticed the comment line at the bottom with the timestamp. Anytime you change anything in our app you'll need to bump that timestamp. Still, other caching mechanisms are in play; so, if for example the manifest file was served with a maxAge header the manifest its self will not be updated until that maxAge elpases. In the mean time the entire site will load from app cache rendering an out of date version. The maxAge state lives with the client and is out of your hands until the maxAge expires or the client clears the cache.
-
Every browser handles app cache a bit different and all of them seem buggy and no one is in a hurry to fix the bugs. It's deprecated after all; so, yeah, offline enabled web apps using app cache, what could go wrong?!
Data Persistence
Application Cache takes care of the web application, storing all the HTML, CSS, and JavaScript files for offline use, but what about data? This is where localStorage and IndexDB come into play.
Local Storage
Local storage is great if your data can be represented by a simple key/value store, but it has some limitations.
-
The spec recommends that localStorage for a single site be capped at 5 megs. In reality the cap can vary by browser. Each browser tends to behave differently if you reach the cap. Some prompt the user, allowing them to increase it, others silently fail.
-
Concurrency across tabs is a beast that has never been an issue in JavaScript before, but get ready because he slumbers no more. LocalStorage does not support a way to both read and update a value as a single atomic operation, but all tabs pointed at your site will share a single localStorage instance; so, consider the simple case of incrementing a number.
let i = localStorage.getItem('counter');localStorage.setItem('counter', i + 1);If this is an operation that occurs in the background in response to an event (say connection status changes), there is a chance, although small that two tabs will receive this event at the same time and...
- Tab 1 will read a value for i, lets say 5.
- Tab 2 will do its read ahead of tab 1's set and also get 5
- Then Tab 2 sets the value to 6
- And Tab 1 as well set it to 6
The net result is you get a 6 where you expected a 7. If you want to make the example more dramatic, albeit ludicrous, you can separate the read and write by a long asynchronous task (setTimeout).
let i = localStorage.getItem('counter');
setTimeout(function() {
localStorage.setItem('counter', i + 1);
}, 5000);
While contrived, this is a simplification of a situation I hit working at airSpring Software while using localStorage to queue operations performed while disconnected.
I found a lot of internet advice pointing to (StorageEvents or incorrectly implemented locking mechanisms)[https://stackoverflow.com/questions/22001112/is-localstorage-thread-safe]. StorageEvents are cool and would let you synchronize form data across tabs, but don't solve the particular problem of updating an entry based on its current state. To address this you really need transactions which localStorage does not support (or an implementation that does not depend on incrementing a value, smarter).
IndexDB
Bring out the big guns! To support an offline write queue you need something a bit more heavy duty. IndexDB is a lot more complicated to use but supports transactions and has much larger usage caps.
In the short example below a while loop is added between when we get our value from indexDB and when we set it to simulate some blocking calculations and increase the window for cross tab collision, but because we are now using transactions no puppies will die! Try it, open two browsers, side by side. Click the button in the first tab then within the 3 second interval click the button in the second tab. The first tab will update to show the value incremented by one, while the second tab will step up by two showing that the first transaction was completely applied before our second tab was allowed to read the value (source also available on github).
<html>
<head>
<title>Offline Demo</title>
<input type="text" value="" id="txtValue" />
<button onClick="runUpdate()">Update</button>
</head>
<body>
<script type="text/javascript">
var dbName = 'offlinestorage';
var dbVersion = 10;
var artificialDelay = 3000; // 3 second delay
var txtValue = document.getElementById('txtValue');
var db;
// initialize our database
_initDB(function(err) {
if (err)
return console.warn(err);
console.log('try to update the index');
// open the db to update our index
var open = indexedDB.open(dbName, dbVersion);
open.onsuccess = function(event) {
console.log('database initialized');
db = open.result;
/* read the current value and
update the textbox with it
*/
readValue(open.result, function(err, val) {
if (err)
return console.warn('err: ' + err);
txtValue.value = val;
});
//_afterInitialization(open.result);
};
});
// ==== Helpers ====
function _initDB(complete) {
var openReq = indexedDB.open(dbName,
dbVersion);
openReq.onupgradeneeded = function() {
console.log('onupgradeneeded');
var db = openReq.result;
if(!db.objectStoreNames.
contains(dbName)) {
console.log('creating the ' +
'objectstore for the first time');
db.createObjectStore(dbName);
}
db.onsuccess = function() {
complete();
};
db.onerror = function() {};
};
openReq.onsuccess = function() {
console.log('done creating objectstore');
complete();
};
openReq.onerror = function() {
complete(
'error initializing the database');
};
}
function readValue(db, complete) {
var transaction = db.transaction([dbName],
'readwrite');
// === get the index and upate it ===
var objectStore =
transaction.objectStore(dbName);
var i = 0;
var getCounterRequest =
objectStore.get('counter');
getCounterRequest.onsuccess = function() {
var iObj = getCounterRequest.result;
if (iObj)
i = iObj;
complete(null, i);
};
}
function incrementValue(db, complete) {
console.log('_afterInitialization called');
var transaction = db.transaction([dbName],
'readwrite');
// === get the index and upate it ===
var objectStore =
transaction.objectStore(dbName);
var i = 0;
var getCounterRequest =
objectStore.get('counter');
getCounterRequest.onsuccess = function() {
var iObj = getCounterRequest.result;
console.log('getConterRequest.result: ' +
iObj);
if (iObj)
i = iObj;
console.log('wait for delay: ' +
artificialDelay);
var dt = Date.now();
do {
/* The old Seinfeld trick,
a loop about nothing
*/
} while(Date.now() - dt < artificialDelay);
/* purposely delay the write
to prove a point
*/
var putCounterRequest =
objectStore.put(i + 1, 'counter');
putCounterRequest.onsuccess = function() {
console.log('put request success');
};
// === transaction left scope and is complete ===
transaction.onerror = function(event) {
complete('error opening transaction');
};
transaction.oncomplete = function() {
console.log(
'HOLY HELL WE UPDATED THE INDEX to ' +
(i + 1) + '!'
);
complete(null, i + 1);
};
};
}
// ===== Event Handlers =====
function runUpdate() {
if (!db)
return console.warn(
'the database was not initialized'
);
/* increment the value and update
the text box
*/
incrementValue(db, function(err, val) {
if (err)
return console.warn('error: ' + err);
txtValue.value = val;
});
}
</script>
</body>
</html>
Only problem, did I say "short"?! That is about as concise as I could keep that and all it does is incrementing a value. I'm not even dealing with browser implementation discrepancies or error handling (tested with chrome 61).
Another problem is that you don't have complete control over when a transaction ends. A transaction ends when it leaves scope. In my example the delay between reading the value and setting back the updated state was generated by blocking code within the same stack frame, but if we had to wait on an async event, such as querying a database in response to the value we found stored in indexDB...
setTimeout(function() {
var putCounterRequest = objectStore.put(i + 1, 'counter');
putCounterRequest.onsuccess = function() {
console.log('put request success');
};
}, artificialDelay);
=> Uncaught DOMException: Failed to execute 'put' on 'IDBObjectStore': The transaction has finished.
Beautiful!
If you need to deal with a situation where the value of an IndexDB entry drives an asynchronous request which leads back to a mutation of the IndexDB value state, then the best solution I've come up with is locks implemented with IndexDB. Unlike attempts I've seen to do locking using LocalStorage these locks can be made thread safe by doing the read and write within a single transaction. Once you have the locking in place you can even use something simpler like localStorage or IndexDB wrapped with localForage for the actual persistence (Sample of this coming in future post). This will work, but holy hell are you sure you want to go down this road!
Where Does this Leave Us?
Building an offline website in 2017 is possible, but comes with considerable overhead and requires attention to detail and careful use of shared resources. Consider which browsers you need to support, how dynamic your site is, what your data representations will look like on the server, and how you'll deal with server side persistence. The more data you accept from users and the more complicated your data storage model the harder this will be. If you have to support all the browsers floating around out this might not be possible at all yet.
Technologies on the horizon like web services, will hopefully help alleviate some of the pains of app cache, but if you walk through web service examples you'll notice, while they give you more control (and more complexity), you run into some of the same problems. When the service code is changed the new service does not take control of the site until after a refresh; so, you're still stuck potentially serving old code and alerting the user to new versions. For files handled by the service worker you could choose to first pull files from the server then use the cache only as a fallback, but then you'd need to choose reasonable timeouts and offline mode would always be slower as it its constantly waiting for timeouts. There's a good chance you'll opt to rebuild something like app cache's offline first model with all of its problems.
These issues can degrade the experience for all users, introducing all new classes of bugs that will affect both offline and online users and adds a lot maintenance overhead.
While offline support for web apps may be sexy if this is not something you absolutely need and you are not Google or Facebook I highly recommend punting on this feature.
If you do really need offline support for your app you may want to consider whether it should really be a web app at all. It may be that you want a native application or could leverage something like electron.
There may come a day when the web can safely deliver applications that happily continue after the connection is lost, when all the tools and patterns are available to make such applications easy for mortal developers to create, but in 2017 there be dragons here.
Samples
A sample file which can be loaded in two browsers to see the cross tab issue with localStorage as well as how it can be solved with indexDB is available on github
Contact
I hope this was helpful. If you want to tell me how full of shit I am or tell me about your great offline capable web app, feel free to tweet me @remembersonly