This week, as part of the request from The Office of Science and Technology Policy and the National Economic Council for public comments to provide input into an upcoming update of the Strategy for American Innovation, PeerJ offered a response in answer to the following question:
“Given recent evidence of the irreproducibility of a surprising number of published scientific findings, how can the Federal Government leverage its role as a significant funder of scientific research to most effectively address the problem?”
Reproducibility is critical in science. Without it science is unable to flourish, and scientists are unable to build on the work of others. Aristotle’s dictum that there is ‘no scientific knowledge of the individual’ seemingly holds true today, as much of the research published in the 21st Century is the result of building on, or testing the findings of others.
The term reproducible research refers to the idea that the ultimate product of academic research is the paper, along with the full computational environment used to produce the results in the paper such as the code and data (1). The full academic output can then be used to reproduce the results and create new work based on the research. Alongside reproducibility lies repeatability – the idea that anyone in the same lab can repeat the same experiment using the same methods and specimens. For science to flourish it is imperative that reproducibility and repeatability become the cornerstones.
Science can only advance on the foundation of the trusted discoveries of others. But like any good building project there is a financial cost to laying these foundations. Scientific research is often funded by governments, and other associated funding bodies, all looking to ensure their money is spent optimally. For instance, some recent research published on reproducibility in the field of cancer studies at the MD Anderson Cancer Center (2) points to the statistic that only 41.5% – 45.4% of scientific outputs were actually reproducible by those surveyed. Other research in this area suggests alarmingly lower figures still of 11% (3).
The US government gives around $30 billion every year in science funding through the NIH (4) which is mainly distributed in research grants to academic scientists. If you were to take the lowest reproducibility rate of 11% that potentially could mean up to 89% of this money (over $26 billion) is wasted. As a tax paying member of the general public you would want to ensure that the government is able to plough your hard earned capital into funds that yield results over and above those figures. It is therefore commendable that the Federal Government is looking to address this issue and leverage it’s role as a significant funder of scientific research.
Beyond the practicalities of finance, there is also an interesting dilemma. Since the middle of the 20th century, life science research concepts and technologies have rapidly grown from the discovery that DNA is the blueprint for life to sequencing and synthesizing new life altogether. Technologies like microarrays, mass spectrometry, high-throughput assays and imaging have been developed, making biology a data-rich science. With all these new tools you could reasonably expect that science would become more rigorous and precise, but with the reproducibility crisis it appears that something entirely opposite could be happening.
So how do we ensure that scientists are provided with the right conditions for their work to be reproducible?
The current state of affairs results from a combination of the complex nature of modern scientific research, a lack of accountability for researchers, and the incentives created by a publish-or-perish culture in academia.
For a scientific researcher to disseminate their work they are hugely reliant on scientific publishers. The publishing of scientific research has always had a large part to play in the visibility of research, and ultimately the reproducibility of science. At PeerJ we believe that the more scientific outputs are made available to all, the better it is for science. We would therefore encourage the Federal Government to put more resource into enforcing open access mandates, to ensure scientific research is opened up to all.
PeerJ publishes articles using a Creative Commons CC-BY licence, which means that authors retain their own copyright, while at the same time others can freely copy and reuse the articles without needing to ask for further permission. If a publisher asks an author to sign over copyright then it becomes difficult, expensive, or impossible for others to access the research. Just as we don’t believe in paywalls blocking access to research, nor do we believe in authors being unable to retain full ownership of their work. By being fully CC-BY, authors and readers don’t need to worry about sharing or reusing articles, so everyone benefits and ultimately science flourishes. The challenge facing those authors who do wish to publish through open access licensing is the proliferation of choice. Choice of licence can be a good thing, but only if there is interoperability in these licences. For instance there is not one common standard among OA licences, and the recently released STM OA licences don’t necessarily operate alongside Creative Commons licences (5). We recommend the move towards everyone using one specific interoperable OA licence.
Scientific journals have a significant role to play in encouraging reproducibility in the first place. They can require more descriptive materials and methods sections and provide unlimited space for them, so that other scientists will know exactly how an experiment was conducted and how they can replicate it. At PeerJ we encourage authors to submit relevant data during the review process, and we would encourage the Federal Government to ensure that more scientific publishers are asking their authors to do so when they submit their work to journals. The current incentive structure for authors does not reward the publication of replication studies. At PeerJ we not only encourage this for our authors, but most importantly our publishing platform enables authors to do just that. We recommend that the Federal Government, and all funders, set aside financial commitment for the replication and publication of the work they fund. We also suggest that the Federal Government looks to set up a specific program incentivising authors to make their data, trackable identifiers, and materials available with publication.
We believe in an open and transparent peer review process. Journals need specialized reviewers to ensure that manuscripts for technically or statistically advanced experiments are vetted thoroughly prior to publication. PeerJ harnesses the talent of thousands of reviewers able to bring their scientific expertise to bear on assessing the science behind the article. But unique to other scientific publishers we encourage our peer reviewers to provide their name as part of their review; and we also give our authors the option to publish the full peer review history of their article alongside the published version. We are hopeful that as more and more journals allow this, and as more and more authors and reviewers experience it, it will become a standard feature of all journals. Ultimately, the reason for doing this is to improve the process of review and publication and to provide fresh new insights for readers. We would ask that the Federal Government consider encouraging and rewarding those publishers which practice some form of open peer-review.
Authors should also be in a position to publish more negative results – those in which an experiment had no effect or clear outcome – because the lack of a finding can sometimes be as important as a finding itself. As technology enables cloud-based storage of all data and file types we encourage authors to openly share their negative results through open data platforms in order that others may learn from the outcomes of their experiments. We would ask that the Federal Government supports those researchers in making their negative data openly available to the world, perhaps by making the reporting of negative (as well as positive) results a requirement of funding.
Scientists are in the privileged position of being able to shape the world for the benefit of mankind, nature and our planet’s future. As outlined, reproducibility and repeatability are the cornerstones for building on scientific discovery and making breakthroughs that help make the world a better place. Without it scientists can’t learn from the work of others, or indeed ensure their own work leaves a legacy to others. It is up to the publishers of scientific research to ensure we do everything we can to provide the best ecosystem for this. It is up to our governments to foster the right environment for that to happen, and reward those who contribute to engendering this.