Offline data synchronization, Part 1: Basic strategies to address this critical challenge for mobile apps
With the ever-growing complexity of modern mobile applications, the connection to powerful servers is crucial. Cognitive elements, speech and image recognition, chatbots, and geodata information for maps are virtually impossible to use without online connections to servers with large data storage and high computational power capabilities. This makes offline data synchronization a critical feature.
For most people, reliable network coverage is almost always available which is convenient since many of their apps rely on constant online access. However, many businesses need their employees to do their work even if the online connection is only available sometimes or not at all. For example, emergency work needs to be done in the case of an electricity blackout or other disaster scenarios when the mobile network might be down or blocked due to heavy use. Some other examples are that workers might be at remote locations like underground construction sites or offshore wind parks without network coverage. In these cases, working with offline data and intelligent data synchronization mechanisms is necessary and is often the most challenging aspect of building mobile solutions for such business environments.
There are many techniques available for data synchronization for mobile solutions. However, it’s crucial to select the right technique for your special business needs. In many projects, clients usually expect the ideal solution: all data is always available offline, never outdated and synchronized smoothly in the background for every possible app function. Unfortunately, this usually conflicts with technological limitations such as the sheer mass of data that needs to be synchronized or limitations in computational power for an efficient processing of complex data synchronization logic.
To have a successful offline data synchronization strategy you need to investigate and understand your specific business requirements in order to choose the right techniques. There is no one-size-fits-all solution. The following are typical scenarios and recommendations for how to handle them:
1. Mass of data
A major core problem for offline use of apps is the sheer amount of data that is required. Workers might need geodata for large regions, thousands of work orders or complex assets consisting of hundreds of parts. Obviously, data reduction is key to this, and this will be discussed later. However, it’s often unavoidable that gigabytes of data need to be stored for offline use.
In such cases, it needs to be investigated when and how often the users have access to a high-speed data connection and which slots in their work schedules are appropriate for downloading the data. The better the connection, the shorter these slots can be.
In one scenario, workers needed to be offline the entire day. They gathered in their offices in the morning to plan and coordinate their work and put on their workwear. So, we decided to use this time slot of approximately 15 minutes to update the data on the device. A high-speed WLAN connection has been established at the office so that up to one gigabyte of data can be downloaded at this time. In other scenarios, data updates happen overnight at the users’ homes through their private internet connection.
To resolve the issue of handling large amounts of data, it’s not only required to look into performance optimization and data reduction but also to look into the business processes and workplace configuration. You need to consider how to best integrate the time slot to download the amount of data needed by the users in their daily or weekly work schedules. Additionally, at key spots, a high-performance network connection might need to be made available or established to provide the necessary bandwidth for the downloads.
2. Sync cycles and prioritization
A crucial topic to investigate is how often offline data is updated and how updates are prioritized. Ideally, all data is always up-to-date. However, in real life, this is rarely guaranteed. Different chunks of data would need to be prioritized based on business needs. Some of the data might only be updated once a year, while other data requires updates several times a day, so different synchronization cycles can be defined for different chunks of data based on business rules. The smaller the packages of data that require high-frequency updates, the better.
For the client, a decision about which data requires only low-frequency updates might be difficult. A helpful tool is the use of forced ranking for the different chunks of data. As this forces the business owner to make a ranking, it significantly helps open the discussion and find compromises between business needs and the technical feasibility of data synchronization.
3. Delta sync: Preprocessed versus on-demand
Next, take a look at the back end. The data that will be synchronized between the mobile device and back-end systems usually needs to be prepared, especially if only a delta between the current data on the mobile device and the server is provided — rather than the full data. Calculating such a delta can be a highly complex task for the server. Processing time can be significantly higher than the time required to exchange the data. This becomes even worse if a large number of clients request individual deltas at the same time from the server. Although on-demand and individual delta calculations might look like elegant solutions at first glance, their applications can often be limited in practice.
A better approach can be preprocessing deltas by the server that are shared with all clients. For example, the server could preprocess a delta for a chunk of data overnight based on the updates of the past working day. In the morning, every client could request a copy of this delta file in order to get updated with the past day’s changes. Again, this requires a deep and thorough understanding of the client’s business needs and limitations to define an appropriate preprocessing strategy.
Another concept to consider besides preprocessing is replication. This comes into play when access and delta processing capabilities of the back-end systems are limited. In such cases, the data can be replicated to a dedicated data store in the mobile middleware. So, the data is prepared or preprocessed for the mobile client without disturbing the core back-end systems. Though this approach provides more flexibility and independence for the mobile application, it comes with new challenges inherited from data replication.
So far, I’ve introduced some basic strategies to handle offline data synchronization. In my next article, “Offline data synchronization, Part 2: Advanced strategies to address a crucial challenge for mobile apps,” I will investigate some more advanced strategies that cover topics such as modifying and updating offline data as well as handling shared data.