Lightweight data management with dtool

Our motivations

Our data management challenges occur at the John Innes Centre (JIC), an independent research institute in plant and microbial sciences. Like many research institutions, the JIC has a strongly decentralised structure and culture. The 40+ research groups act mostly as independent units.

This poses a significant challenge to any data management process, and renders many existing solutions, which rely on enforced compliance with centralised systems, difficult to use.

This also tends to lead to situations where key metadata are encoded in file names and paths, metadata which can easily be lost when files are moved. Coupled with concern about actual data loss/corruption when moving files, this leads to reluctance to archive data. As a result, storage systems become full and data accumulates.

The JIC has a mixture of different storage technologies bought at different times. Each technology has its own quirks with which the end user needs to gain familiarity. For example, access to data on a file system and object-based storage work differently, user and group management varies from system to system and some technologies provide the ability to access deleted files while others do not. Having to manage different storage systems is not a productive use of researcher’s time.

We needed a solution that would:

• Provide clear, immediate benefit to the front line data managers, such as core facility staff or bioinformaticians embedded in research groups.

• Allow group leaders and institute management to get an overview of the data they have.

• Enable use of different storage systems and technologies, without changing tools and pipelines.

The solution needed to be easy to use to get users started without a long learning process. It also needed to avoid lengthy migration of data to centralised platforms.

These are all common requirements for those managing data in heterogeneous research environments. Therefore any solution that meets these needs is likely to be valuable to a wide range of researchers and support groups, particularly those without existing centralised data management systems.

Here we describe our lightweight approach to managing data. It centers around the concept of packaging metadata with data, and working with the two as a unified, portable whole.

Packaging data and metadata into a unified whole

Our solution to this data management problem is dtool. It is lightweight in that it has no requirements for a (central) database. It simply consists of a command line tool and an application programming interface (API) for packaging and interacting with data.

The most important aspect of dtool is that it packages data files with accompanying metadata into a unified whole. The packaged data and metadata is referred to as a dataset. Having the metadata associated with the data means that datasets can be moved around and organised. It also means that the dataset contains all the information required to verify the integrity of the data within it.

To illustrate the benefits of packaging data and associated metadata into a unified whole, it is worth comparing it to other alternatives. A common solution is to store metadata in file names and directory structures. For example consider a file named col0_chitin_leaf_1.tif stored in a directory named repl_2. The file name contains several pieces of metadata, namely that the image is of leaf sample 1 (leaf_1), of the Colombia-0 ecotype of A. thaliana (col0), treated with chitin (chitin). Furthermore the information that this is replicate 2 (repl_2) is encoded in the directory structure. This makes it hard to move this data around without accidentally losing metadata, since this metadata is implicitly encoded in the file path, rendering this approach fragile.

Another common approach is to store metadata in a database, this is the solution used by systems such as iRODS and openBIS. A database is quite a heavyweight solution for managing metadata. It has the disadvantage that one needs access to the database to be able to work with the data, making it difficult to work off-site when the database is managed centrally within an institute. It also makes it difficult to move data into other systems.

When dtool is used to create a dataset, it generates both administrative metadata and structural metadata (Fig. 1). The administrative metadata contains information that helps manage the dataset and includes, for example, an automatically generated universally unique identifier (UUID). The structural metadata describes how the dataset is put together, for example each data item in the dataset has associated information about its size in a manifest, stored as part of the dataset.

When creating a dataset the user is asked to add descriptive metadata about the dataset (Fig. 2). The user is, for example, prompted to describe the dataset, state the project name and whether or not the dataset contains any confidential or personally identifiable information.

Design decisions

The structure of the dataset was designed to be able to outlive the tool used to generate it. In practise this means that metadata files are plain text and make use of standard file formats such as JSON and YAML. It also means that there are files dedicated to describing the structure of the dataset itself (.dtool/README.txt and .dtool/structure.json in Fig. 1).

The structural metadata in the manifest (.dtool/manifest.json) was designed to be able to verify the content of a dataset and abstract away file paths. It therefore stores the size, checksum, and relative path of each item in the dataset. Below is the content of a sample manifest.

 
 
 { 
  ”dtoolcore˙version”: ”3.7.0”, 
  ”hash˙function”: ”md5sum˙hexdigest”, 
  ”items”: { 
    ”8bda245a8cd526673aab775f90206c8b67d196af”: { 
      ”hash”: ”78608288625e084d68b4b5e8c1725933”, 
      ”relpath”: ”ERR022075˙2.fastq.gz”, 
      ”size˙in˙bytes”: 1982742623, 
      ”utc˙timestamp”: 1537805410.0 
    }, 
     ”9760280dc6313d3bb598fa03c5931a7f037d7ffc”: { 
      ”hash”: ”4766e60e72da987e4c54e47fee5653a4”, 
      ”relpath”: ”ERR022075˙1.fastq.gz”, 
      ”size˙in˙bytes”: 1875702420, 
      ”utc˙timestamp”: 1537805358.0 
    } 
  } 
}    

dtool was designed to abstract away the underlying storage system. The structure of a dataset on a file system is illustrated in Fig. 1. The representation in other storage systems, such as object storage, can be different. However, the end user’s interactions with dtool remain the same (Fig. 2), no matter what the underlying storage system is.

To ensure stability, the core code base (dtoolcore) has no dependencies outside of the Python standard library. The core code base has a high test coverage (91%) to minimise the chance of accidentally introducing bugs when developing new features. The high test coverage is a result of insisting on tests for all core code that relate to the creation and interaction with datasets. Tests were implemented using the pytest package and coverage was measured using CodeCov.

The dtool storage backends and the command line interface make use of a pluggable architecture, which makes it possible to write separate Python packages for creating custom storage backends and adding new sub-commands to the dtool command line interface without touching the existing code bases.

Language choice

The dtool API and command line tool were implemented in Python. Python was chosen as it is open source, freely available and well established in the scientific community with a rich ecosystem of scientific packages. It was also chosen because it is the language the authors prefer to program in.

Source code and documentation

The dtool source code is freely available under the liberal MIT licence on GitHub. The architecture of the code is pluggable with a core available at https://github.com/jic-dtool/dtoolcore.

The dtool documentation was created using Sphinx and is hosted using Read the Docs. It is available at http://dtool.readthedocs.io/.

Example scripts in the Results section and the supplementary material are available at https://github.com/jic-dtool/dtool_examples.

Use case: making sense of data

One of the challenges in starting work in a new lab is coming to grips with old lab members’ data. The person who generated the data is often no longer around and substantial effort can be spent trying to understand the context of the data and the way it has been structured.

dtool enables researchers to understand the context and content of a dataset by packaging the metadata with the data. In other words one can quickly get from a URI specifying the location of a dataset to an overview of the dataset. The URL below represents a dataset hosted in Amazon S3 that can be accessed by anyone using dtool (the URL is not intended to be displayed using a web browser).

 
 
http://bit.ly/Ecoli-reads    

To find out the name of this dataset one can use the dtool name command.

 
 
$ dtool name http://bit.ly/Ecoli-reads 
Escherichia-coli-reads-ERR022075    

To find out the UUID of this dataset one can use the dtool uuid command.

 
 
$ dtool uuid http://bit.ly/Ecoli-reads 
faa44606-cb86-4877-b9ea-643a3777e021    

In the examples above dtool pulls out the name/UUID of the dataset and prints it to the terminal.

To get more information about this dataset one can use the dtool readme show command.

 
 
$ dtool readme show http://bit.ly/Ecoli-reads 
--- 
description: Whole Genome Sequencing of Escherichia coli str. K-12 MG1655 
design: Paired-end sequencing (2x100 base) of E. coli library 
sample: E. coli K-12 strain MG1655 
study: — 
  Paired-end sequencing of the genome of Escherichia coli K-12 
  strain MG1655 
  using the Illumina Genome Analyzer IIx 
Library: 
  Name: CT1093 
  Instrument: Illumina Genome Analyzer IIx 
  Strategy: WGS 
  Source: GENOMIC 
  Selection: RANDOM 
  Layout: PAIRED 
  Construction protocol: — 
    Standard Illumina paired-end library construction protocol. 
    Genomic DNA was 
    randomly fragmented using nebulisation and a ˜600 bp fraction 
    (including adapters) was obtained by gel electrophoresis. 
links: 
  - SRA: https://www.ncbi.nlm.nih.gov/sra/ERX008638 
  - ENA: https://www.ebi.ac.uk/ena/data/view/ERX008638    

The command above pulls out the descriptive metadata from the dataset and prints it to the terminal. In this case the descriptive metadata tells us, amongst other things, that this dataset contains paired-end sequencing data for E. coli K-12 strain MG1655.

To see the size of the dataset one can use the dtool summary command.

 
 
$ dtool summary http://bit.ly/Ecoli-reads 
name: Escherichia-coli-reads-ERR022075 
uuid: faa44606-cb86-4877-b9ea-643a3777e021 
creator˙username: olssont 
number˙of˙items: 2 
size: 3.6GiB 
frozen˙at: 2018-09-26    

This reveals that the dataset contains two items and is 3.6GiB in size. The items in the dataset can be listed using the dtool ls command.

 
 
$ dtool ls --verbose http://bit.ly/Ecoli-reads 
8bda245a8cd526673aab775f90206c8b67d196af   1.8GiB  ERR022075˙2.fastq.gz 
9760280dc6313d3bb598fa03c5931a7f037d7ffc   1.7GiB  ERR022075˙1.fastq.gz    

In the above the -v/–verbose flag is used to return the size as well as the identifier and the relative path of each item.

In summary the commands dtool readme show, dtool summary and dtool ls give an overview of the context and content of a dataset.

Use case: backing up raw data and archiving old data

At the JIC we have several storage solutions, each one serving a specific purpose. Relatively expensive file system storage is used for processing data. Object storage accessible via the S3 API, with off-site backups is used for storing raw data. A capacious storage system front-ended by iRODS is used for archiving long term intermediate data. Because dtool abstracts away the underlying storage solution the end users can use the same commands for copying data to and from these differing storage systems. The ease of moving data around can be illustrated by copying a dataset hosted in the cloud to local disk.

 
 
$ dtool cp -q http://bit.ly/Ecoli-ref-genome . 
file:///Users/olssont/Escherichia-coli-ref-genome    

In the above the -q/–quiet flag is used to only return the URI specifying the location that the dataset has been copied to, in this case a directory named Escherichia-coli-ref-genome in the current working directory.

dtool can be used to copy a dataset between different storage solutions. This enables researchers to copy data to storage solutions designed for backing up and archiving data.

Use case: generating inventories of datasets

One of the challenges of running a research group is keeping track of all the data being generated. As such it is useful to be able to list datasets and to generate inventories of datasets. This can be achieved using the commands dtool ls and dtool inventory.

The purpose of dtool ls is to provide a way to list names and URIs of datasets. Below is an example of the dtool ls command listing three datasets stored in a directory named my_datasets (see the supplementary material for details on how to setup this directory).

 
 
$ dtool ls my˙datasets 
Escherichia-coli-reads-ERR022075 
  file:///Users/olssont/my˙datasets/Escherichia-coli-reads-ERR022075 
Escherichia-coli-reads-ERR022075-minified 
  file:///Users/olssont/my˙datasets/Escherichia-coli-reads- 
ERR022075-minified 
Escherichia-coli-ref-genome 
  file:///Users/olssont/my˙datasets/Escherichia-coli-ref-genome    

The need for this command becomes more apparent when working with datasets stored in the cloud. The command below lists datasets in the Amazon S3 bucket dtool-demo. Note that the command below requires the user to have permissions to read the bucket and as such will not work for the readers of the paper, but is included for illustrative purposes.

 
 
$ dtool ls s3://dtool-demo/ 
Escherichia-coli-ref-genome 
  s3://dtool-demo/8ecd8e05-558a-48e2-b563-0c9ea273e71e 
Escherichia-coli-reads-ERR022075-minified 
  s3://dtool-demo/907e1b52-d649-476a-b0bc-643ef769a7d9 
Escherichia-coli-reads-ERR022075 
  s3://dtool-demo/faa44606-cb86-4877-b9ea-643a3777e021    

The dtool inventory command is intended to be able to generate reports of datasets. The command below creates a report (my_datasets.html) listing all the datasets in the my_datasets directory as a single HTML file that can be shared with colleagues.

 
 
$ dtool inventory --format=html my˙datasets ¿ my˙datasets.html    

In summary the dtool ls command can be used to find data in a base URI and dtool inventory can be used to generate reports and web pages to make datasets findable.

Use case: verifying the integrity of old data

It is useful for researchers to be able to reassure themselves that their research data is intact.

In order to be able to check whether or not this is the case, dtool provides a means to verify the integrity of a dataset, using the dtool verify command. By default this checks that the expected item identifiers are present in the dataset and that files have the correct size. There is also a further option to check that the content of the files is correct by comparing checksums, using the -f/–full option.

 
 
$ dtool verify Escherichia-coli-ref-genome 
All good :)    

The command above shows that the dataset contains the expected content. To illustrate what happens if a dataset becomes corrupted we can move a file out of the dataset.

 
 
$ mv Escherichia-coli-ref-genome/data/U00096.3.fasta . 
$ dtool verify Escherichia-coli-ref-genome 
Missing item: b445ff5a1e468ab48628a00a944cac2e007fb9bc U00096.3.fasta    

In summary dtool provides a means to get clarity with regards to the integrity of a dataset. This is not, however, intented to detect deliberate tampering with the dataset, i.e., it would be possible to make the dtool verify command pass by deliberate modification of the underlying files and the manifest.

Use case: processing data

dtool provides programmatic access to the data in a dataset. This means that one can use dtool to create scripts that abstract away the location of the data.

For example, to process all the items in a dataset one can use the dtool identifiers command to list all the identifiers. To access the content of the items one can then use the dtool item fetch command, which returns the absolute path to a location where the item can be read. For datasets stored in the cloud, the dtool item fetch implicitly includes a step to download the item to local disk to ensure it can be read from the absolute path returned by the command.

Below is a Bash script (simple_processing.sh) to illustrate this. The script extracts the first line from each dataset item, using gunzip and head.

 
 
#!/bin/bash 
 
# Read in the input dataset URI from the command line. 
INPUT˙DS˙URI=$1 
 
# Process all the items in the input dataset. 
for ITEM˙ID in ‘dtool identifiers $INPUT˙DS˙URI‘; do 
    # Fetch an item and process it. 
    ITEM˙ABSPATH=‘dtool item fetch $INPUT˙DS˙URI $ITEM˙ID‘ 
    gunzip -c $ITEM˙ABSPATH — head -n 1 
done    

Running this simple_procssing.sh script on a dataset stored in the cloud is illustrated below.

 
 
$ bash simple˙processing.sh https://bit.ly/Ecoli-reads-minified 
@ERR022075.1 EAS600˙70:5:1:1158:949/2 
@ERR022075.1 EAS600˙70:5:1:1158:949/1    

We can verify that this gives the same results as running the script on a dataset stored on local disk by copying the dataset and re-running the script on the local dataset.

 
 
$ LOCAL˙DS˙URI=‘dtool cp -q https://bit.ly/Ecoli-reads-minified .‘ 
$ bash simple˙processing.sh $LOCAL˙DS˙URI 
@ERR022075.1 EAS600˙70:5:1:1158:949/2 
@ERR022075.1 EAS600˙70:5:1:1158:949/1    

It is also possible to use dtool to store the output of processing scripts, both in terms of data and metadata. In other words, it is possible to implement scripts that implement dataset to dataset processing. This can be used to automate aspects of data management.

The script below, called minfiy.sh, uses this concept of dataset to dataset processing. It is worth noting that the script:

• Creates a name for the output dataset based on the input dataset name

• Creates an output dataset

• Processes all the items from the input dataset and adds the results to the output dataset

• Extracts the descriptive metadata from the input dataset as a base for the descriptive metadata of the output dataset

• Adds a reference to the input dataset and a description of how it was processed to the descriptive metadata of the output dataset

 
 
#!/bin/bash 
 
# Exit immediately on failure of a command. 
set -e 
 
# Read in the input from the command line. 
INPUT˙URI=$1 
OUTPUT˙BASE˙URI=$2 
NUM˙LINES=4000 
 
# Create a name for the output dataset based on the input dataset. 
OUTPUT˙NAME=‘dtool name $INPUT˙URI‘-minified 
 
# Create an open proto dataset. 
OUTPUT˙URI=‘dtool create -q $OUTPUT˙NAME $OUTPUT˙BASE˙URI‘ 
 
# Process all the items in the input dataset and 
# add the results to the output dataset. 
for ITEM˙ID in ‘dtool identifiers $INPUT˙URI‘; do 
 
  # Fetch the item from the dataset and get an absolute path 
  # from where its content can be accessed. 
  ITEM˙ABSPATH=‘dtool item fetch $INPUT˙URI $ITEM˙ID‘ 
 
  # Write the minified version of the item to a temporary file. 
  TMP˙MINIFIED=$(mktemp /tmp/minfied.XXXXXX) 
  gunzip -c $ITEM˙ABSPATH — head -n $NUM˙LINES — gzip ¿ $TMP˙MINIFIED 
 
  # Add the temporary file to the output dataset giving it the relpath 
  # of the item from the input dataset. 
  RELPATH=‘dtool item relpath $INPUT˙URI $ITEM˙ID‘ 
  dtool add item $TMP˙MINIFIED $OUTPUT˙URI $RELPATH 
 
  # Cleanup. 
  rm $TMP˙MINIFIED 
done 
 
# Create descriptive metadata for the output dataset. 
TMP˙README=$(mktemp /tmp/dtool-readme.XXXXXX) 
dtool readme show $INPUT˙URI ¿ $TMP˙README 
echo ”minified:” ¿¿ $TMP˙README 
echo ”  from˙UUID: ‘dtool uuid $INPUT˙URI‘” ¿¿ $TMP˙README 
echo ”  from˙URI: $INPUT˙URI” ¿¿ $TMP˙README 
echo ”  content: first $NUM˙LINES per item” ¿¿ $TMP˙README 
                                                                                                                                        
# Add the descriptive metadata to the output dataset. 
dtool readme write $OUTPUT˙URI $TMP˙README 
 
# Cleanup. 
rm $TMP˙README 
 
# Finalise the output dataset. 
dtool freeze $OUTPUT˙URI    

In Supplemental Information 1 there is a script that performs a Bowtie2 (Langmead & Salzberg, 2012) alignment. It takes as input a dataset with paired RNA sequencing reads, a dataset with a reference genome and a base URI specifying where the output dataset should be written to. The command below shows the usage of this script.

 
 
$ bash bowtie2˙align.sh  “ 
  http://bit.ly/Ecoli-reads-minified  “ 
  http://bit.ly/Ecoli-ref-genome .    

Running this command creates a dataset in the current working directory. Below is a command to create an environment variable as a simple alias to this directory.

 
 
$ DS˙URI=Escherichia-coli-reads-ERR022075-minified-bowtie2-align    

The content of this dataset is a SAM (Sequence Alignment Map) file.

 
 
$ dtool ls $DS˙URI 
3ffaeaf15fc1f12417aadddb9617fb048e39509e  ERR022075.sam    

The descriptive metadata gives information about how this SAM file was derived.

 
 
$ dtool readme show $DS˙URI 
--- 
description: bowtie2 alignment 
input˙reads˙uri: http://bit.ly/Ecoli-reads-minified 
ref˙genome˙uri: http://bit.ly/Ecoli-ref-genome 
bowtie˙version: bowtie2-align-s version 2.3.3    

It is important to note that the metadata above was generated automatically by the bowtie2_align.sh script.

In summary, dtool provides a means to write processing scripts that are agnostic to where the input data is stored, whether it be on local disk or in some object storage system in the cloud. Furthemore, using dtool to store the data generated from processing scripts allow researchers to automate parts of their data management.

Use case: sharing data

It is possible to share datasets hosted in cloud storage such as Amazon S3 and Microsoft Azure storage.

Take for example the dataset represented by the URI below.

 
 
s3://dtool-demo/8ecd8e05-558a-48e2-b563-0c9ea273e71e    

This is the dataset with the E. coli reference genome data.

 
 
$ dtool name s3://dtool-demo/8ecd8e05-558a-48e2-b563-0c9ea273e71e 
Escherichia-coli-ref-genome    

This URI can only be used by people that have been authorised to interact with the dtool-demo Amazon S3 bucket. To make this dataset accessible to the public one can use the dtool publish command.

 
 
$ dtool publish -q  “ 
  s3://dtool-demo/8ecd8e05-558a-48e2-b563-0c9ea273e71e 
https://dtool-demo.s3.amazonaws.com/8ecd8e05-558a-48e2-b563-0c9ea273e71e    

It is now possible for anyone in the world to interact with this dataset using the HTTPS URI returned by the dtool publish command.

 
 
$ dtool name  “ 
  https://dtool-demo.s3.amazonaws.com/8ecd8e05-558a-48e2-b563-0c9ea273e71e 
Escherichia-coli-ref-genome    

To make life easier one can use a URL shortner like Bit.ly to create a more user friendly URI. The example below refers to the same dataset as above.

 
 
dtool name http://bit.ly/Ecoli-ref-genome 
Escherichia-coli-ref-genome    

In summary dtool makes it possible to share datasets with collaborators and to make datasets accessible to the research community.

Uptake at JIC

At the JIC we have introduced twelve research groups that deal with high volumes of data (>10TB) to dtool. Of these, five groups have adopted dtool to manage their data, another four are still evaluating it and three have found it difficult to move away from their existing working practises.

Of the three groups that have found it difficult to move away from their exising working practises the main barriers to uptake have included: (1) feeling overwhelmed with the amount of exising data; (2) tight coupling of data analysis and input files, i.e., hard coded paths in scripts; (3) use of directory structures to encode provenance of data analysis; (4) absence of a data management champion within the group.

Four of the five groups that have adopted dtool found the incentive to do so in association with a key member leaving the group. The other group that adopted dtool did so as part of starting a new project. In this instance we helped the group devise a protocol for packaging raw sequencing data into datasets as it arrived from the sequencing company.

To date 26TB of data has been packaged into 803 datasets.