Potential bias in peer review of grant applications at the Swiss National Science Foundation
- Published
- Accepted
- Subject Areas
- Science Policy
- Keywords
- Research funding, Peer review, Swiss National Science Foundation, Grant peer review, Research evaluation, Gender bias, SNSF, Switzerland, Grant proposal, Public research funding
- Copyright
- © 2019 Severin et al.
- Licence
- This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ Preprints) and either DOI or URL of the article must be cited.
- Cite this article
- 2019. Potential bias in peer review of grant applications at the Swiss National Science Foundation. PeerJ Preprints 7:e27587v2 https://doi.org/10.7287/peerj.preprints.27587v2
Abstract
Background: The Swiss National Science Foundation (SNSF) supports fundamental and use-inspired research in all academic disciplines. As part of the evaluation procedure, grant applications to the SNSF are reviewed by external reviewers. The legitimacy of funding decisions depends on its ability to base funding decisions solely on the scientific merit of grant applications.
Aim: We examined whether the following factors influenced the scores given to grant applications submitted to the SNSF: (1) source of nomination of the reviewer, (2) the gender of the applicant and the reviewer, and (3) the country of affiliation of the reviewer.
Methods and Data: We gathered data on 38,250 external reviews of 12,294 unique grant applications across all disciplines between 2006 and 2016. Proposals were rated on a scale from 1 (=poor) to 6 (=outstanding) by 26,836 reviewers. We used linear mixed effects regression models adjusted for research topic, applicant’s age, nationality and affiliation.
Results: We found that in univariable analysis applicant-nominated reviewers awarded higher evaluation scores than reviewers nominated by the SNSF. Further, reviewers affiliated with research institutions outside of Switzerland gave more favourable evaluation scores than reviewers affiliated with Swiss institutions. Finally, male reviewers awarded higher evaluation scores than female reviewers and male applicants received more favourable evaluation scores than female applicants. When we controlled for confounding factors, adjusted differences changed little for source of nomination and country of affiliation. In contrast, the gender differences nearly disappeared, which indicates that most of the gender effects observed in univariable analysis is explained by differences in scores between research topics and applicant affiliations.
Conclusions: Our study showed that peer review of grant applications at SNSF may be prone to biases stemming from different applicant and reviewer characteristics. Based on this study the SNSF abandoned nomination of reviewers by applicants, and made members of panels aware of the other systematic differences in scores. We encourage other public funding bodies to conduct similar studies.
Author Comment
We spotted a small number of copy-paste mistakes in Table 1 and Table 2 (supplementary files), which we corrected in version 2 of the preprint. These amendments have not changed the overall findings and conclusions.
Supplemental Information
Table 2. Unadjusted and adjusted differences in scores assigned by external reviewers of grant applications submitted to the SNSF
Notes. Unadj. Effect = unadjusted, crude effect; Adj. Effect = adjusted effect; SE = standard error; 95% CI = 95% confidence interval; *p < .05 (t-test. t-values and degrees of freedom reported online at www.git.io/fhaJx).
Figure 5: Average evaluation scores by research topic for female and male applicants and proportions of female and male applicants by research topic
Upper panel: Horizontal lines indicate Wald confidence levels, with the number of reviews for each point.
Figure 6: Average evaluation scores by age group for female and male applicants and proportions of female and male applicants per age group
Upper panel: Horizontal lines indicate Wald confidence levels, side by side with the number of reviews for each point.
Figure 7: Average evaluation scores by type of affiliation for female and male applicants and proportions of female and male applicants by affiliation
Upper panel: Horizontal lines indicate Wald confidence levels, side by side with the number of reviews for each point.
Figure 8: Average evaluation scores by nationality for female and male applicants
Horizontal lines indicate Wald confidence levels, side by side with the number of reviews for each point.
Figure 9: Average evaluation scores by application call deadline
Horizontal lines indicate Wald confidence levels, side by side with the number of reviews for each point.