Going Beyond Flat File Data on IoT Devices with Embedded Database

IoT endpoint devices are responsible for so much information, IoT software developers need to carefully consider how data will be managed and stored. Managing data collected on sensors, gateway devices, and embedded systems is a complex task, especially over a long period of time. Data is everything in modern development practices, with many convenient ways to manage, access, and share information. Yet ARM embedded software developers still use flat files to store information in market-dependent formats that are difficult to efficiently analyze and communicate to other systems.

Flat files may be adequate when there is only a small volume of data and no need for local reporting. However, when specific information needs to be searched and analyzed, flat files are not a viable solution. In many cases, it isn't possible to proactively filter data before it is written to the file system. Only when enough data is accumulated to understand the full context can the software decide which records are most valuable. Unexpected failures often occur with IoT devices, and if flat files are used for more than simple logging, data corruption is a real possibility that can cause system breakdown.

As a method of data management, flat files have an advantage when data is only logged and never analyzed or filtered on a device. While some analysis is possible with small files on a device, performance quickly becomes an issue when it becomes necessary to relate records that were stored in different files or at different times. Duplicating data in different files can help, but maintaining consistency across multiple files quickly becomes a redundancy nightmare.

Database tables allow incoming information to be divided into multiple related records that are easy to search in a variety of ways, with or without SQL. When database software is integrated into an IoT endpoint device, those queries and decisions can be made right on the device, without transferring every record to a back-end data management system first. In this way, the device can be more selective and timely with the information it shares with the Cloud and various web services.

Without database transactions, every non-trivial change to the file system carries a small risk of data loss or inconsistency, especially when an unexpected power failure clears the device's working memory. Storing records in ITTIA DB SQL database tables mitigates this risk and enables applications to search over a large volume of data at the same time.

Flat files also do not support multiple user or multi-threaded systems limiting access to important data when multiple users or threads are searching for the same data, at the same time. When concurrent tasks share access to the storage media, it is important to never overwrite information that other tasks might be actively reading. Developers typically define critical sections to protect shared resources, using mutual exclusion to completely block shared access while a resource is being used. However, critical sections must be applied uniformly and should execute quickly, both difficult conditions to guarantee with flash media. ITTIA DB SQL automatically protects tables with database transactions, which allow concurrent tasks to safely and efficiently share access.

Other ITTIA Benefits include:

• The ITTIA DB SQL embedded database is part of an end-to-end data communication solution.
• ITTIA support and services make device software development a breeze.

When data is shared continuously with web services or other devices, sending the same records more than once is very inefficient. Tracking the status of each record individually introduces significant overhead, but how else can you account for updated or deleted records? ITTIA DB SQL uses replication logging to track changes sequentially, enabling efficient bi-directional synchronization with other relational databases.

Check out ITTIA DB SQL, our cross-platform embedded database product.

Learn more about overcoming the limitations of flat files in the white paper: Breaking the Flat File Barrier for Embedded Developers

10-01-2017