Preserving Transactional Data

Preserving Transactional Data. Sara Day Thomson. DPC Technology Watch Report 16-02. May 2016.
     This report examines the requirements for preserving transactional data and the challenges in re-using these data for analysis or research.   Transactional will be used to refer to "data that result from single, logical interactions with a database and the ACID properties (Atomicity, Consistency, Isolation, Durability) that support reliable records of interactions."

Transactional data, created through interactions with a database, can come from many sources and different types of information. "Preserving  transactional data, whether large or not, is imperative for the future usability of big data, which is often comprised of many sources of transactional data.  Such data have potential for future developments in consumer analytics and in academic research and "will only lead to new discoveries and insights if they are effectively curated and preserved to ensure appropriate reproducibility."

The organizations who collect transactional data aim to manage and preserve collected data for business purposes as part of their records management. There are strategies for database preservation as well as tools and standards  that can look at data re-use. The strategies for managing and preserving big transactional data must adapt to both SQL and NoSQL environments. Some significant challenges include the large amounts of data, rapidly changing data, and different sources of data creation. 

Some notes:

• understanding the context and how the data were created may be critical in preserving the meaning behind the data
• data purpose: preservation planning is critical in order to make preservation actions fit for purpose while keeping preservation cost and complexity to a minimum
• how data are collected or created can have an impact on long-term preservation, particularly when database systems have multiple entry points, leading to inconsistency and variable data quality.
• Current technical approaches to preserving transactional data primarily focus on the preservation of databases. 
• Database preservation may not capture the complexities and rapid changes enabled by new technologies and processing methods 
• As with all preservation planning, the relevance of a specific approach depends on the organization’s objectives.

There are several approaches to preserving databases:

• Encapsulation
• Emulation 
• Migration/Normalization

• Archival Data Description Markup Language (ADDML)
• Standard Data Format for Preservation (SDFP) 
• Software Independent Archiving of Relational Databases (SIARD)

"Practitioners of database preservation typically prefer simple text formats based on open standards. These include flat files, such as Comma Separated Value (CSV), annotated textual documents, such as Extended Markup Language (XML), and the international and open Structured Query Language (SQL)." The end-goal is to keep data in a transparent and vendor-neutral database so they can be  reintegrated into a future database.

Best practices:

1. choose the best possible format, either preserving the database in its original format or migrating to an alternative format.
2. after a database is converted, encapsulate it by adding descriptive, technical, and other relevant documentation to understand the preserved data.
3. submit database to a preservation environment that will curate it over time.

Research is continuing in the collection, curation, and analysis of data; digital preservation standards and best practices will make the difference between just data and "curated collections of rich information".

Digital Preservation Matters.