Reviewers rarely comment on the same aspects of a manuscript, making it difficult to properly assess manuscripts’ quality and the quality of the peer review process.
“My biggest hope is that editors will consider implementing structured peer review in their journals and that they will publish results of those trials”
For All Readers - AI Explainer
Why is peer review important in academic publishing?
Peer review ensures the quality and credibility of academic manuscripts by having experts evaluate the work before it is published.
What problem does this research address?
Reviewers often focus on different aspects of a manuscript, making it hard to uniformly assess both the manuscript’s quality and the effectiveness of the peer review process.
What was the goal of this study?
The study aimed to evaluate the implementation of structured peer review by examining how reviewers responded to specific questions, the level of agreement between reviewers, and whether structured questions improved the review process compared to traditional methods.
What is structured peer review?
Structured peer review involves asking reviewers to answer a set of predefined questions about various aspects of a manuscript.
How was the pilot study conducted?
In August 2022, nine structured questions were piloted in 220 Elsevier journals. A random 10% of these journals were selected, and manuscripts that received two reviews within the first two months were analyzed, totaling 107 manuscripts from 23 journals.
What types of questions were included in the structured peer review?
Eight questions required open-ended responses, and one question about language editing had a yes/no option. Reviewers could also provide comments to the author and editor.
How were the responses analyzed?
Two independent raters used qualitative methods to analyze the reviewers’ answers.
Did reviewers respond to the structured questions?
Yes, 92% of reviewers provided answers to all the structured questions, despite them not being mandatory.
What was the nature of the responses?
The longest responses (median of 27 words) were for questions about reporting methods in enough detail for replication. The highest agreement (72%) was for manuscript flow and structure, while the lowest agreement was for data interpretation and statistical analyses (around 52-53%).
How did structured peer review affect traditional review comments? Two-thirds of the reviewers filled out the Comments-to-Author section, and about half of these comments resembled traditional peer review reports, covering multiple topics from the structured questions.
Was there an improvement in reviewer agreement on final recommendations?
Yes, the absolute agreement on final recommendations was 41%, an improvement from the 31% agreement observed between 2019 and 2021.
What are the main takeaways from this study?
The structured peer review format was well-received by reviewers, leading to more comprehensive coverage of topics. There was improved agreement on final recommendations, though individual questions about data interpretation and statistical analyses showed significant disagreement.
What are the study’s limitations?
The study wasn’t a randomized trial, so further research is needed to confirm the findings and to explore whether structured peer review improves knowledge transfer or manuscript quality.
What future research is suggested?
Additional studies should be conducted to verify the results and to determine the broader impacts of structured peer review on the peer review process and manuscript improvement.
What are Article Spotlights?
Spotlighted articles are press released, and feature author interviews, AI explainers and more.
If you have published in Peer J and would like to be featured in an Article Spotlight please contact PeerJ.