Impact is more than just a number. For this interview series, we revisited some of the most cited papers published in PeerJ Computer Science and asked the authors to tell us about the real-world impact and applications of their research beyond the number of citations they garnered.
In the second interview of the series Daniel S. Katz, Arfon M. Smith and Kyle E. Niemeyer, co-authors of Software citation principles, discuss the impact of the paper on recognising the mostly invisible software work that is currently unrewarded in the academic and research environment.
First, we’re delighted to hear that our paper is one of the ten most-cited PeerJ CS articles! We believe this paper is an important starting point in ongoing community work that is intended to make the mostly invisible work done by software developers and maintainers into work that is visible and rewarded within the academic and research environment.
Background
We wrote this paper as the output of a community activity as part of FORCE11. At the time that we started, it had become clear to us and to many others that software was an essential part of research, but that the developers and maintainers of this software, particularly in the open source community, were not receiving academic recognition for their work, and that this was due to the fact that there were no standard ways to do so across the various stakeholder communities, including the researchers themselves, but also their institutions, publishers, indexers, librarians and archivists, funders, and professional societies. The first step in making what is, in essence, a cultural change was to define a relatively simple set of principles with buy-in from these stakeholders.
Our charge included, “The group will produce a set of principles, illustrated with working examples, and a plan for dissemination and distribution,” and this is what we did. Over about 2 years, we brought together about 55 people, and starting with the recently published data citation principles, we updated our ideas based on software use cases and related work, working group discussions, community feedback, and an in-person workshop, leading to the PeerJ CS article. And it’s also important to mention that the fourth author of the paper is the overall FORCE11 working group.
Impact of the paper and work
This paper has helped raise the profile of software as a key element of research, tied to a number of different community activities including the UK Software Sustainability Institute and the international Research Software Engineering movement. More focused examples include the recent Research Data Alliance’s Covid-19 Guidelines and Recommendations, which includes a section on software, which in turn including a recommendation to cite software that references the PeerJ CS paper, and inclusion of software in the draft revision of the OECD Committee for Science and Technology Policy’s “Enhanced Access to Publicly Funded Data for Science, Technology and Innovation,” the final version of which will become soft law for OECD member states.
Following this work, a new FORCE11 Software Citation Implementation Working Group was created to take these principles and move them forward. This group has written checklists for authors and developers, defined remaining challenges, and has active task forces working on metadata, repositories, and journals (including publishers and conferences).
The difficulty in citing software led us and others, led by Arfon, to create the Journal of Open Source Software as a method to establish a straightforward process for this aimed at developers. The journal is now publishing over 400 papers per year, giving developers something that can be cited to help their contributions to research be recognized.
This paper and work has also impacted all of our careers, perhaps most notably in Kyle specifically mentioning this highly cited article (along with its high altmetrics score) in the candidate statement of his dossier applying for promotion to Associate Professor and tenure, when discussing service activities and the Software Citation Working Group. Though it is unclear how much of a role this played in the grand scheme of things, the application was successful.
About us
Daniel S. Katz
Dan is Assistant Director for Scientific Software and Applications at the National Center for Supercomputing Applications (NCSA), Research Associate Professor in Computer Science, Electrical and Computer Engineering, and the School of Information Sciences (iSchool), and Faculty Affiliate in Computational Science and Engineering at the University of Illinois Urbana-Champaign. Dan’s interest is in the development and use of advanced cyber infrastructure to solve challenging problems at multiple scales. His technical research interests are in applications, algorithms, fault tolerance, and programming in parallel and distributed computing, including HPC, Grid, Cloud, etc. He is also interested in policy issues, including citation and credit mechanisms and practices associated with software and data, organization and community practices for collaboration, and career paths for computing researchers. He is a senior member of the IEEE and ACM, a co-PI of the US Software Sustainability Institute (URSSI) Conceptualization project, a co-founder of the US Research Software Engineer Association, a co-founder of the Research Software Alliance, and a founding editor and current Associate Editor-in-Chief of the Journal of Open Source Software.
Arfon M Smith
Arfon has had a career developing research software in a variety of academic settings: he co-founded the Zooniverse citizen science platform at The University of Oxford (2008) and spent the next 5 years developing a scalable generalized platform for citizen science projects which ultimately led him to the Adler Planetarium in Chicago as Director of Citizen Science. In 2013 he joined GitHub Inc., the world’s largest platform for open source software, to support academic users of the platform. This led to a number of significant improvements, including support for rendering Jupyter Notebooks natively on GitHub and integrations with services such as Zenodo and figshare for automatically assigning DOIs for GitHub repositories. During his time at GitHub he also co-chaired the FORCE11 Software Citation Principles Working group (one of the outputs of which being this PeerJ paper). In May 2016, he launched the Journal of Open Source Software, which is on target to publish its 1000th paper in 2020. Over the last four years, the community demand for JOSS has only grown: the editorial board is now over 50-strong (from an initial group of 15) and semi-annual calls for editors receive a large number of highly-qualified candidates. In November 2016 he moved to the Space Telescope Science Institute in Baltimore to establish the first Data Science Mission Office for the Institute but has continued in his capacity as JOSS Editor-in-Chief during this appointment.
Kyle E Niemeyer
Kyle is an Associate Professor of Mechanical Engineering in the School of Mechanical, Industrial, and Manufacturing Engineering at Oregon State University. His research focuses on using computational modeling to understand combustion and reacting fluid flows, and developing numerical methods particularly for modern heterogeneous computing systems. As a result, he and his research group use and develop quite a bit of research software, and he is an advocate for open-source software in research and open science more generally. Kyle is a founding editor and current Associate Editor-in-Chief of the Journal of Open Source Software and the Journal of Open Source Education.
Publishing with PeerJ
We decided to publish this paper with PeerJ CS because we strongly wanted it to be open access, and one of us (Dan) was already a PeerJ CS Academic Editor, so familiar with the goals of the journal, including attempting a different mechanism for publishing to change the culture of the community while maintaining high standards, which aligned with the goals of our work. PeerJ also let us publish a preprint to build up some community interest, which helped us better develop and write the final paper, and the fact that the preprint and paper system were integrated was attractive to us. PeerJ CS also allowed us to put in appendices that add to the understanding of the paper. Finally, the on-line and PDF versions of the paper are visually pleasing and easy-to-read.
The review and publication process was reasonably quick, and was fair. We had good reviewers who encouraged us to improve some part of the paper, and we also appreciate the fact that we were able to publish the review history, with the review names and the discussion between us and them as part of the record of the paper.
One amusing thing that happened during our PeerJ CS review process is that we had a paragraph in our initial draft that said that software was different than data from the point of view of citation, and gave a few examples of why this was the case, using this as a motivation to explain why we couldn’t just use the data citation principles but needed to develop something new. One reviewer disagreed with us making this statement, and asked that we instead cite some published work. This led us to form a small group to discuss this, which we did via GitHub, using a Markdown document to capture our thinking and GitHub issues to discuss potential additions and changes. This process led to a better set of differences that in the initial paragraph, and we published this as a PeerJ preprint, back when that was still possible, and then cited this preprint, which satisfied the reviewer.
Thank you for sharing your research journey, Dan, Arfon and Kyle. You can read their paper here
Submit your next article to PeerJ Computer Science here
You can find more PeerJ author interviews here. View related research on Software Engineering