Authors’ response: the reaction above by Marjet Elemans points to a number of issues. One of the main issues addressed is the analysis of Dutch universities, possible differences between what is reported here vs. what is reported by universities themselves, and what this indicates as to the quality of data and methodology used here.
Our intention with the section on Dutch data is not to correct or supplant the OA counts of Dutch universities. We chose the Netherlands because of the reasons stated in the paper: the size of this subset of data, Dutch OA policies and our knowledge of the Dutch national context. To our knowledge there is no open data available on OA levels as reported by universities themselves, other than the reported overall percentage of OA for all universities (42%), which is in line with our overall finding for the Netherlands (although the figures cannot be directly compares as our Dutch figure includes non-university output). It would be very welcome to be able to corroborate the data on different types of OA with data provided by universities themselves especially if those are also accompanied by full descriptions of methods used.
Any such comparison should differentiate carefully between differences caused by definitions and inclusion criteria (i.e. whether green OA includes submitted versions (before peer review), whether disciplinary repositories are included in addition to institutional repositories, whether bronze OA is included) and differences caused by data quality, (i.e. representation of research output in Web of Science and coverage and quality of harvesting of institutional repositories by oaDOI). The latter are acknowledged as caveats in the paper and should indeed be kept in mind when interpreting the data, the former are a question of definition, not of correct vs. incorrect data.
Below, we address the other points made one by one.
- regarding the lack of a definition of usability:
We agree that we need to elaborate on usability when we mention that as a goal and will do so in a next version of the paper. For now it may suffice that when we say usability we refer to issues as transparency of the data, units of analysis that make sense for real-world questions/situations, enough diversity that comparisons are meaningful, the degree to which counts can be reproduced, the ease with which counts can be generated, the universality of the applicability regarding countries, fields, institutions, and finally the costs involving generating the counts. Added to that, the usability would in the end not be up to us to judge. It would be the degree to which the WoS/oaDOI data delivers insights considered helpful by OA stakeholders themselves. Our exploration can help them make that judgment.
- regarding the lack of focus without testable hypotheses:
The absence of hypotheses is a direct consequence of choosing to do exploratory, hypotheses- generating research. The reason we take this approach is exactly because this is new and uncertain data presented in a new way. As stated in our discussion section hypothesis testing could be done in a later stage. The present article could inform those subsequent efforts
- regarding caveat 2:
On the potential effects of a time lag between WoS and oaDOI updates Elemans states that we could quantify that effect by by comparing the results of the OA tool in WoS with the results of oaDOI. However, that is not possible. The only thing that would check for is the current time lag of WoS update of oaDOI data, and the results would depend on where in the update cycle of oaDOI and WoS the moment of measurement lies.
- regarding caveat 3:
Technically it would indeed be possible controlling for the effect of inclusion of the ESCI sub-database, and perhaps we will do that exercise. However, it is important to note that (1) coverage of WoS is changing constantly even without addition of ESCI and (2) we are testing the usefulness of full WOS core, in the way most people will use it. We also addressed the effect of ESCI in some more detail in the sections on language and countries.
- regarding caveat 4:
For the wish to zoom in on subcategories of gold OA (pure gold, hybrid and bronze): this is exactly why we made the extra effort to discern between the various types of OA using the dutch case. We are not entirely sure what it is that would be lacking in our paper in this regard. There are several reasons why gold and hybrid can change with time: not in the sense that a gold or hybrid paper will lose that status over time, but in the sense that possibilities & circumstances for publishing gold and hybrid change continuously (deals, funds, APC levels, etc.). The levels found for the past publication years could in part reflect these kind of changes.
Further, regarding the analyses of longitudinal data called for: strictly speaking we do not have longitudinal data: the data we use should be regarded as a snapshot. That is the point caveat 4 makes. We did look at the data for a range of publication years broken down to OA type, which is what the available data allow.
- Regarding caveat 5:
oaDOI does not explicitly state which version of DOAJ is used and how often they update this. As oaDOI data is meant to reflect current, not historical OA availability, it is presumed that checking is done against one, current version of DOAJ, not separate versions from each publication year. DOAJ does provide information on the first calendar year journal provided online Open Access content - it is not known whether oaDOI uses this information, but this would be indeed good to verify with them.
Regarding the effect of journals removed from DOAJ: if the articles are still in Web of Science and are freely available at the publisher website, they would show up as bronze or hybrid (depending on licensing data in Crossref). If the journals have been removed from WoS, they would not show up in our analysis at all (from the time of removal, if older articles remain). This could cause a decrease in pure gold levels (and corresponding increase in bronze and/or hybrid OA) or possibly a decrease in levels of overall OA, respectively. Whether that would be a detectable difference compared to current levels would depend on the number of articles in such journals in any given set. In this respect, it is good to note that Web of Science can add or remove journals, including OA journals, at any time, and inclusion in DOAJ is likely just one factor in this.
- Regarding caveat 6:
For each unit of analysis, thresholds used in selection (total number of articles & reviews in the period 2010-2017) are indicated in the methodology section, where applicable. No additional threshold was used for the number of articles & reviews in a given publication year. All absolute numbers are available in the accompanying dataset.
……….
Finally: Elemans mentions the lack of attention to data quality. We are very much aware of issues in the data, which is exactly why we formulated the caveats and why we welcome the feedback and continue to discuss these matters with data providers.
