How To Become Good At Peer-Review
The following guest blog post, by Jennifer Raff, has been adapted for PeerJ from a post she made on the Violent Metaphors blog. We are pleased to host this guest post, as we feel that it provides a great overview for young (or not so young!) researchers who might be asked to conduct a peer-review for a journal such as PeerJ.
Peer review is at the heart of the scientific method. Its philosophy is based on the idea that one’s research must survive the scrutiny of experts before it is presented to the larger scientific community as worthy of serious consideration.
Reviewers (also known as referees) are experts in a particular topic or field. They have the requisite experience and knowledge to evaluate whether a study’s methods are appropriate, results are accurate, and the authors’ interpretations of the results are reasonable. Referees are expected to alert the journal editor to any problems they identify, and make recommendations as to whether a paper should be accepted, returned to the authors for revisions, or rejected. Referees are not expected to replicate results or (necessarily) to be able to identify deliberate fraud.
While it’s by no means a perfect system (see, for example, the rising rates of paper retractions), it is still the best system of scientific quality control that we have. In fact, it’s such a central part of the scientific process that one can often identify questionable scientific findings by their authors’ objections to the rigor of review (and attempts to circumvent it by a variety of methods, including self-publication).
Yet the quality of this system is ultimately dependent on the quality of the reviewers themselves. Many graduate programs don’t explicitly teach courses on how to review papers. Instead, a young scientist may learn how to review a paper under the guidance of his or her mentor, through journal clubs, or simply through trial and error. I’ve been thinking about this topic a lot, and decided to put together a set of guidelines for young scientists. In doing so, I also hope to help non-scientists understand a bit more about the process.
Here are some important practices to keep in mind as you conduct the review:
1. Expertise: Ask yourself honestly whether this paper falls within the scope of your expertise. If it falls too far outside of your discipline and knowledge, it’s better to leave the review to someone else. If you decline to review (for whatever reason), it’s courteous to do so promptly (else you risk delaying the process) and to recommend several alternate referees to the editor.
2. Timeliness: It’s important that you make an effort to return a review within the specified deadline. People’s research careers may be depending on how quickly their papers are published. Take into consideration your schedule when you’re deciding whether to review a paper. If you’re a relatively young researcher, you may need to put in additional time in order to familiarize yourself with any background literature and methods that might be new to you. This isn’t a small thing, especially when balanced with all your other duties (coursework, teaching, research, writing). And keep in mind that complicated papers take more time to evaluate. For example, it took me about 8 or 10 hours (spread out over several days) to do my most recent review of a particular manuscript, and another 4 hours to review the revised version. If you don’t think you can commit enough time to the review in order to make the deadline, you should decline, or request an extension, before accepting the invitation.
3. Take it seriously: Peer review is at the heart of the scientific process, and in order for the process to work, the review must be rigorous. Don’t just sign off on a paper or do a slapdash job; read it critically (see my suggestions for how to do so below). Your review should ideally help the authors to improve the quality of their manuscript, and contribute to the overall quality of the journal.
4. Avoid bias: Absolutely do not review a manuscript if you have a strong feeling (positive or negative) about any of the authors. Absolutely do not review a manuscript with any intention of reaping personal or professional benefit from it. Absolutely do not review a manuscript if you have any possible conflict of interest in doing so (for example, having collaborated with those authors in the past). If in doubt, declare your possible conflict before agreeing to the review.
Be aware that several studies have revealed implicit biases (such as gender bias) in peer review. Studies have reported that abstracts with female senior authors are systematically more poorly reviewed than abstracts with male senior authors (Knobloch-Westerwick et al. 2013), and that “all articles with women in dominant author positions receive fewer citations than those with men in the same positions.” (Larivière et al. 2013)
To avoid biases of all kinds, make a concerted, deliberate effort to always review the manuscript in front of you, not the authors or their location.
5. Don’t be intimidated: Remember that reviewing is a place where the implicit hierarchy of academia should not apply, ever. The authors of the manuscript might be the biggest names in the field, but that doesn’t automatically make them right.
6. Respect confidentiality: Don’t talk about the manuscript, its results, or its methods with outsiders. Don’t use information from the manuscript prior to its publication. Don’t communicate with the authors about their manuscript. All thoughts and comments on it should only go to the editor.
How to review critically, but fairly
You should be an expert at this by now, because you’ve been reading and thinking about papers nonstop since the beginning of graduate school. The process is basically the same for reviewing, except that a reviewer must be even more thorough. Here are a few tips, divided into four topics (if you want some more suggestions on how to read critically, my post on how to read and understand the scientific literature applies to scientists as much as non-scientists). I suggest you take notes constantly as you go through the review process. Draft your response only after you’ve read the entirety of the submission (including Supplementary Online Information) several times! Absolutely avoid being unnecessarily harsh or in any way abusive; your criticism should be constructive in tone – a peer-review is not the place to demonstrate your wit or to disrespect authors in any way.
As you go through your review, make sure to reference the specific parts of the manuscript that you are referring to (for example, by line number).
1) The Introduction/Background
Some questions to ask:
An approach to critically evaluating the Introduction/Background portion of a manuscript.
Things to watch out for in this section:
– An irrelevant study. Ask yourself whether the questions the authors are addressing are really advancing the field in a meaningful way. This does not mean that an article has to be completely novel, but it does mean that the work contributes to the sum of knowledge in the field and does not, for example, simply repeat well known results.
– A paper which does not include all appropriate references, or which inappropriately emphasizes some. I suggest identifying the specific question and doing a literature search on your own, as if YOU were writing the paper. Then compare your list of references to the authors’. Are you familiar with all of them? (This approach has tangential benefits, as it will help you become better acquainted with the literature of your field). If the authors cite certain studies with which you’re not familiar, read them! See if you agree with how they’re discussed. If the authors haven’t cited studies that they should have, make a note of it, and include it in your review. And yes, it’s perfectly fine to suggest that they cite your own papers, if it’s appropriate (the flip side: don’t be “that reviewer” who demands that everyone cite him/her, or agree with his/her pet hypotheses. This process isn’t about you).
For journals such as PeerJ and PLOS ONE, the emphasis is on scientific and methodological soundness , not potential impact or readership (See PeerJ’s guidelines for authors and reviewers here). Some questions to ask:
An approach to critically evaluating the Approach/Methods portion of a manuscript
The authors should have provided enough detail in both the methods and materials they have used, such that you can fully understand their experiments (and that others can replicate them). If they don’t, ask them to do so.
Don’t hesitate to read outside sources on how to conduct an experiment or how other labs use them. Don’t hesitate to ask other people for help understanding methods in a general sense (while still respecting the confidentiality of the manuscript). You need to be confident in your assessment of whether the authors did the experiments correctly. If you’re not, you can tell the editor privately and request an additional expert to review that portion of the manuscript.
Keep in mind that while you have every right to ask the authors to do a different experiment, or to modify the way they carry out an existing one, this should be justified from the point of view of verifying the results reported. Remember that doing so will take time, effort, and money, so don’t do this lightly. (And remember that under no circumstances should you even think of doing this in order to slow a lab down so that your project can be published first. That is EXTREMELY unethical and can be grounds for a formal complaint.).
3) Interpreting the research
Some questions to ask:
An approach to critically evaluating the Results/Interpretation/Discussion portion of a manuscript
Absolutely feel free to ask for more results, or better ways of presenting them. If you believe that the data require the review of a statistical expert (or another kind of expert), recommend it to the editor. If you believe material in the Supplementary Online Information section would be more useful in the main body of the paper (or vice versa), recommend it!
4. Some final considerations
By this point, you will probably have read the entirety of the paper several times. Is the writing clear and free of grammatical errors? If not, request additional editing before publication (it isn’t necessarily your job to point out specific grammatical errors, though you are free to do so).
Have you read any associated statements, such as the Funders Statement, the Ethics Statement, and Conflict of Interest statement, and were they reasonable? Have the authors specified a mechanism by which they will make raw data from their experiments available? Have they specified how they will protect the identity of their human subjects (particularly a concern in genomics research)? Are there any additional ways in which the paper could be improved? Include these considerations in your write up to the editor.
Anonymity vs. open review
There have been calls for doing away with anonymity of referees, as anonymity is seen by some as facilitating bullying. In fact, the British Medical Journal (BMJ) requires all reviewers to be identified. In a study of the effect of open peer review on the quality of reviews, it was suggested that “if reviewers have to sign their reviews they may put more effort into their reviews and so produce better ones “, and also that “ open peer review should increase both the credit and accountability for peer reviewing, both of which seem desirable.” (van Rooyen et al. 1999). In contrast, this discussion in Nature Neuroscience recommended the continuation of anonymous review. As an early career scientist, I feel more comfortable critiquing my colleagues’ work anonymously. But I know several peers who have reached the opposite conclusion.
In order to improve transparency PeerJ encourages, but does not require, open review and (at the option of the authors) they post the entire history of reviewers’ and editors’ comments along with the paper. The aim is to facilitate dialogue between reviewers, editors, and the author in order to improve the overall quality of the paper. One useful aspect of this policy is that new reviewers can now study the peer-review process of specific articles, and learn from the ways in which that process was conducted. Another useful benefit of providing your name, is that you can now gain public credit for having performed reviews. Ultimately, however, the decision as to whether or not to sign your review is a completely personal one.
There has been a growing movement towards post-publication peer-review—community commenting on aspects of a paper after it has been published—in order to focus assessment of scientific impact on the quality of the paper itself than the prestige of the journal in which it appeared. Post-publication peer-review maintains that the publication of scientific findings is the beginning of their evaluation, not the end. Experimental replication, and evaluation of the study by the broader scientific community are also important components of the process. While several journals — for example PeerJ or PLOS ONE — have taken steps towards allowing comments on their own papers, a different approach has been to establish third party locations to facilitate comments about any published article – for example, by the likes PubPeer, which allows any paper to be commented upon anonymously, and PubMed Commons, which allows any PubMed-indexed paper to be commented upon non-anonymously by PubMed-indexed authors. In this way, anyone should feel enabled to act as a peer-reviewer (albeit after publication) regardless of whether or not they were invited to do so before publication.
As you can see, reviewing can be a very difficult task, but you will definitely improve with practice! Journal clubs are an excellent way to improve your critical reading skills, as well as help from your mentor. Browse the comments on PeerJ, PubPeer and PubMed Commons to see how other scientists approach reviewing a paper. I’ve also found that the best way of improving my review abilities is to actually go through the process as an author. Remember that (minus the odd antagonistic person), referees are there to help you improve your work. Listen to and learn from them! The scientific community demands a lot from our reviewers. Reviewing manuscripts may not give you obvious immediate benefits, but understand that doing so is a good way of keeping abreast of cutting edge research in your discipline. Going through the process makes you better at critical reading and thinking, and allows you to give back to your field in a very important way. I view it as a sort of civic obligation to the broader scientific community. Your contribution as a referee helps keep the scientific process as fair as possible.
Many thanks to Dr. Elizabeth Wager for her helpful thoughts and resources on this subject.
References and useful resources
Peer review: the nuts and bolts: http://www.senseaboutscience.org/pages/peer-review-the-nuts-and-bolts.html
Nature’s peer review debate: http://www.nature.com/nature/peerreview/debate/
Wager et al. 2002. “How to survive peer review” London: BMJ Book. http://www.bmj.com/sites/default/files/attachments/resources/2011/07/wager.pdf
Committee on publication ethics: Guidelines for Peer Reviewers by Irene Hames on behalf of COPE Council, March 2013, v. 1. http://publicationethics.org/files/u7140/Peer%20review%20guidelines.pdf
COPE’s new Ethical Guidelines for Peer Reviewers: background, issues, and evolution: http://www.ismte.org/Shared_Articles-COPEs_new_Ethical_Guidelines_for_Peer_Reviewers_background_issues_and_evolution
van Rooyen et al. 1999. Development of the Review Quality Instrument (RQI) for Assessing Peer Reviews of Manuscripts. Journal of Clinical Epidemiology 52(7) 625-629. (http://www.sciencedirect.com/science/article/pii/S0895435699000475)
Nicholas, KA and Gordon, WS. 2011 “A Quick Guide to Writing a Solid Peer Review” EOS 92 (28): 233-240. (http://onlinelibrary.wiley.com/doi/10.1029/2011EO280001/pdf)
Parberry, I. “A guide for new referees in theoretical computer science.” http://www.cs.utexas.edu/~dahlin/professional/parbery-referee.pdf
Moher D and Jadad AR. “How to peer review a manuscript” http://www.bmj.com/sites/default/files/attachments/resources/2011/07/moher.pdf
Nature’s checklist for authors: http://www.nature.com/authors/policies/checklist.pdf
PRISMA’s resources for systematic reviews and meta-analyses: http://www.prisma-statement.org/
STROBE’s resources for epidemiology studies: http://www.strobe-statement.org/index.php?id=available-checklists
Nature Neuroscience’s guide for reviewers (http://www.nature.com/neuro/referees/index.html), and “Striving for excellence in peer review” (http://www.nature.com/neuro/journal/v12/n1/full/nn0109-1.html)
Petrovečki, M. The role of statistical reviewer in biomedical scientific journal. Biochemia Medica 2009;19(3):223-30. http://dx.doi.org/10.11613/BM.2009.020
Gould, T. H. P. (2010). Scholar as e-publisher: The future role of [anonymous] peer review within online publishing. Retrieved from http://krex.ksu.edu
van Rooyen et al. 1999. Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ 318 pp 23-27
Wager et al.2006. Are reviewers suggested by authors as good as those chosen by editors? Results of a rater-blinded, retrospective study. BMC Medicine 2006, 4:13 (http://www.biomedcentral.com/1741-7015/4/13)
Please feel free to suggest additional resources, or your thoughts on how to improve this post in the comments below, or by emailing me at jenniferraff (at) utexas (dot) edu. Thanks in advance!