Making computer science results reproducible - A case study using Gradle and Docker
Author and article information
Abstract
This paper addresses two questions related to reproducibility within the context of research related to computer science. First, requirements on reproducibility are analyzed based on a survey addressed to researchers in the academic and private sector. The survey indicates a strong need for open but also easily accessible results, thus reproducing an experiment should not require too much effort. The results from the survey are then used to formulate general guidelines for making research results reproducible. In addition, this paper explores a number of existing software tools that could bring forward reproducibility in research results. After a general analysis of tools a further investigation is done via three case studies based on actual research projects which are used to evaluate the previously introduced tools. Results indicate that due to conflicting requirements, none of the presented solutions fulfills all intended goals perfectly. However, we present requirements and guidelines for making research reproducible. While the main focus of this paper is on reproducibility in computer science, the results of this paper are still valid for other fields using computation as a tool.
Cite this as
2018. Making computer science results reproducible - A case study using Gradle and Docker. PeerJ Preprints 6:e27082v1 https://doi.org/10.7287/peerj.preprints.27082v1Author comment
This is a submission to PeerJ Computer Science for review.
Sections
Supplemental Information
Responses from online survey on reproducibility in software-based research and engineering
Responses from online survey on reproducibility in software-based research and engineering.
Additional Information
Competing Interests
The authors declare that they have no competing interests.
Author Contributions
Wilfried Elmenreich conceived and designed the experiments, performed the experiments, analyzed the data, contributed reagents/materials/analysis tools, performed the computation work, authored or reviewed drafts of the paper, approved the final draft.
Philipp Moll conceived and designed the experiments, performed the experiments, analyzed the data, contributed reagents/materials/analysis tools, performed the computation work, authored or reviewed drafts of the paper, approved the final draft.
Sebastian Theuermann conceived and designed the experiments, performed the experiments, analyzed the data, contributed reagents/materials/analysis tools, performed the computation work, authored or reviewed drafts of the paper, approved the final draft.
Mathias Lux conceived and designed the experiments, performed the experiments, analyzed the data, contributed reagents/materials/analysis tools, prepared figures and/or tables, performed the computation work, authored or reviewed drafts of the paper, approved the final draft.
Data Deposition
The following information was supplied regarding data availability:
We have added links to the used tools directly into the paper. This includes data and code produced by us as well as code from others where we have built upon.
The following repositories contain parts deposited for the submitted paper:
Docker Images:
https://hub.docker.com/r/phmoll/saf-prebuild/
https://hub.docker.com/r/dermotte/liresolr/
https://hub.docker.com/r/frodewin/frevo/
used version of FREVO software, Sourceforge:
Funding
This work was supported in part by the Austrian Science Fund (FWF) under the CHIST-ERA project 496 CONCERT (project no. I1402). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
