Gender and other potential biases in peer review: Analysis of 38,250 external peer review reports
Author and article information
Abstract
Background The Swiss National Science Foundation (SNSF) supports fundamental and use-inspired research in all disciplines. Peer reviewers assess the proposals submitted to the SNSF. We examined whether the gender of applicants and reviewers and other factors influenced the summary scores awarded.
Methods We analysed 38,250 reports on 12,294 grant applications across all disciplines 2006 to 2016. Proposals were rated on a scale from 1 (=worst) to 6 (=best) by 26,836 reviewers. We used linear mixed effects regression models adjusted for research topic, applicant's age, nationality, affiliation and calendar period to examine associations, and interactions between gender of the applicant and other variables.
Results In univariable analysis, male applicants received more favourable evaluation scores than female applicants (+0.19 points; 95% CI 0.14-0.23), and male reviewers awarded higher scores than female reviewers (+0.12; 95% CI 0.08-0.15). Applicant-nominated reviewers awarded higher scores than reviewers nominated by the SNSF (+0.53; 95% CI 0.50-0.56), and reviewers affiliated with research institutions outside of Switzerland more favourable scores than reviewers affiliated with Swiss institutions (+0.53; 95% CI 0.49-0.56). In multivariable analysis, differences between male and female applicants were attenuated (to +0.08; 95% CI 0.04-0.13) whereas results changed little for source of nomination and affiliation of reviewers. There was an interaction between gender of applicant and reviewer, and between gender of applicant and calendar period. Male reviewers gave higher scores than female reviewers, with a greater difference for male than for female applicants (P=0.037 from test of interaction). The gender difference increased after September 2011, when new evaluation forms were introduced (P=0.033 from test of interaction).
Conclusions Our study showed that peer review of grant applications at SNSF might be prone to biases stemming from different applicant and reviewer characteristics. The SNSF abandoned the nomination of peer reviewers by applicants, and made members of panels aware of the other systematic differences in scores. The new form introduced in 2011 may inadvertently have given more emphasis to the applicant’s track record, and a revision is now under discussion. We encourage other funders to conduct similar studies, in order to improve the evidence base for rational and fair research funding.
Cite this as
2019. Gender and other potential biases in peer review: Analysis of 38,250 external peer review reports. PeerJ Preprints 7:e27587v3 https://doi.org/10.7287/peerj.preprints.27587v3Author comment
The new version includes the following changes:
- Additional analyses on interaction effects between the different variables of interest (gender, country of affiliation, source of nomination)
- Inclusion of old and new evaluation guidelines/forms for external reviewers of grant applications
Sections
Additional Information
Competing Interests
AS, JM, FD, AJ are employed at the Swiss National Science Foundation. JM is seconded at the European Research Council. ME is the President of the National Research Council of the Swiss National Science Foundation.
Author Contributions
Anna Severin prepared figures and/or tables, authored or reviewed drafts of the paper, approved the final draft, investigation, Project Administration.
Joao Martins conceived and designed the experiments, performed the experiments, analyzed the data, contributed reagents/materials/analysis tools, prepared figures and/or tables, authored or reviewed drafts of the paper, approved the final draft, investigation, Project Administration.
François Delavy analyzed the data, authored or reviewed drafts of the paper, approved the final draft.
Anne Jorstad authored or reviewed drafts of the paper, approved the final draft.
Matthias Egger conceived and designed the experiments, authored or reviewed drafts of the paper, approved the final draft.
Rachel Heyard analyzed the data, prepared figures and/or tables, approved the final draft.
Data Deposition
The following information was supplied regarding data availability:
Associated data is available at Zenodo.
DOI: 10.5281/zenodo.2592509
URL: http://doi.org/10.5281/zenodo.2592509.
Code is available at URL: https://zambujo.github.io/snsf-peerreview/.
The published data is a slightly modified version to ensure that it is anonymized: the information on the date of the call has been deleted, the age variable has been regrouped in 5 year groups and two very special cases have been deleted.
Funding
The authors received no funding for this work.
