To review or not to review
Receiving an invitation to review a manuscript for a journal usually comes by email. The email will contain some details of what’s expected (e.g. the timeframe), the title and abstract of the manuscript, and links allowing you to accept or decline the invitation. If you have been published (and therefore had your own work reviewed), then you should make some effort to give back to the community. But what factors might influence your decision to review a given manuscript?
Time: Most journals ask you to submit your review within 2 or 3 weeks. Not long! Of course, you can ask the editor for extensions, but you should probably avoid accepting an invitation if you will not have time in the next month.
Conflicts of interest: It is essential that you avoid reviewing work if you have a conflict of interest, or some pre-existing bias. This will usually include if you are affiliated with the same department (or even institution), if you are a collaborator (whether or not you’ve published together), if you are friends, or even if you feel predisposed to disagree with the ideas that are being presented. Sometimes you won’t realise that you have a conflict of interest until you’ve already accepted the invitation to review as most journals do not include the authors’ details in the invitation. In such cases, you should write to the editor straight away to clarify this so that the editor can seek other reviewers.
Expertise: The whole idea of peer-review is for research to be independently evaluated by experts. Obviously, you should have some familiarity with the work you are being asked to evaluate. This does not mean it must be exactly what you study, and editors often choose reviewers with varying areas of expertise to ensure the final review incorporates a broader scientific perspective. If you know the background literature or theory, are familiar with the methods, or have experience with the system, then you’re qualified to go ahead with the review. Lastly, the work you are reviewing should be at least partly relevant to your own research or your interests.
As a final note, you should aim to respond to the invitation email promptly (1-2 days). By not responding, you are unnecessarily delaying the authors’ chances of receiving prompt decisions on their work.
What is the review supposed to achieve?
The primary focus of the peer-review process is to ensure that published papers are scientifically valid. Scientific validity can be a big hurdle, and under no circumstances should we lose focus on the importance of this. This can sometimes be a somewhat subjective judgement—so it is important to keep in mind that papers are also part of a broader and more general scientific discourse. As a reviewer, you should focus your efforts on lack of clarity, questionable approaches, circular arguments, or anything that gives you doubts about the validity of the study.
Plan out your reading
Reviewing a manuscript will usually involve more than one read through. It is a good idea to plan ahead for when you’ll do this and what you aim to achieve with each reading. There are several strategies for doing this, but you should usually plan to include three different readings of the manuscript.
The skim read: It is useful to start with having a quick skim read of the work. The aim here is to get a general impression of the work: what is the question and how is the paper organised? Where are apparent the strengths and weaknesses of the work? Do you generally understand what the authors did, and why? Considering these questions will also help you to determine how long will it take to complete this review. The best time to do this is when you first accept the invitation and receive the link to download the manuscript. This will give you a chance to get your mind into the right gear for the topic, and plan ahead for how long you will need, before you go more in-depth with your next read(s).
The major comments read: The most important read when reviewing a manuscript is one in which you try to look beyond typos, grammar, or other presentation issues, and instead focus on the broader strengths and weaknesses of the research. Here you will be looking to address most of what the peer-reviewing process aims to achieve: scientific validity (see more details below). Sometimes it is difficult to keep this in focus if there are many typographical issues with the work, in which case you might want to get these out of the way first (see below). Fundamentally, this read should focus on whether the methods and results satisfactorily address the questions and support the conclusions the authors set out in the (1) title, (2) abstract, (3) introduction, and (4) concluding paragraphs.
The minor comments read: It is always helpful to assist the authors with improving the presentation of their work. This generally includes pointing out mistakes in the text and highlighting when passages in the text are unclear. It can also be helpful to give feedback on the design of figures, or even suggestions for changing the order of paragraphs (this can often happen with the introduction). This read mostly focuses on typographical issues, which we often call the ‘minor comments’ or ‘line by line comments’.
Hints to make reading easier
Getting the first read done early will really help with setting up your expectations of the paper and giving yourself time to think through some of the concepts presented in the paper before you go for your in-depth read. At this point, you might add some question marks on areas that might require you to do some further digging. It can also help to print out the paper, as this allows you to make notes about typos, etc. without having to move your focus away from the text. Printing the paper can facilitate keeping track of the bigger picture, such as by quickly flipping back to the abstract or title, which can be a bit disorienting in a digital document.
Key areas to focus during the major comments read
Methods: There are several reasons why the most likely area to raise problems is the methods section. (i) A published paper should be written with sufficient detail and clarity that someone who is completely unfamiliar with the work can replicate it. While reading this, don’t forget that the authors are intimately familiar with their own work, so what seems crystal clear for them may not be so obvious to others. Never hesitate to push the authors to provide more details or justification. (ii) The reasoning behind the approach used to test the key questions or hypothesis in a paper should be logical. Does the test actually answer the question that the authors laid out? It is surprising how often this is not the case. Science relies on logical hypothetico-deductive reasoning, which centers on testing a relevant and falsifiable null hypothesis. Make sure that the authors have a clear link between their questions and how they test their hypothesis. (iii) Are the statistical methods correct? Again, there is room for as much opinion as hard facts about what is right or wrong, but you should never hesitate to push the authors to justify their statistical decisions. This can include ensuring that they appropriately satisfy the assumptions of the methods that they use, that they have thought about why they are using the method (as opposed to simply copying other papers), and that they do not run the risk of elevated false positives or false negatives (e.g. through multiple hypothesis testing or unnecessarily transforming variables).
Predictions: The general format we use, at least in biology, is to lay out our predictions in the last paragraph or two of the introduction. That is, the introduction sets out the key hypothesis or hypotheses, and then argues through logical construction for predictions that, if true, could be used as evidence in support of the hypotheses. It is essential that these predictions actually do support the hypothesis. The key question is therefore: does the inference from hypothesis to biological relationship necessarily hold true? Very often there are alternative processes, mechanisms, or explanations that could generate similar patterns, and these violate the process of providing support for a hypothesis.
Results: The results section can often be very short, but it should not be overlooked! It is very important that the manuscript includes all of the appropriate details of the statistical results, including the full outputs of the statistical tests performed. Very often papers provide ‘naked’ P values, omitting the other components of the statistical test (e.g. degrees of freedom), and generally completely ignoring effect sizes. It is also useful to look at the statistics and the figures together to see if these match, and if the authors are correctly interpreting their statistical results. For example, if the statistics report a significant positive effect, is this apparent in the relevant figure? It is also crucial to make sure that the authors accurately present their data. Do the figures show the raw data rather than just plotting the model fits (e.g. the predicted values, which can be deceiving)?
Discussion: It is quite important to scientific rigour that results are presented in an honest way. The discussion (and parts of the introduction, abstract, and title) should be carefully reviewed for overselling or hyping up. Even more importantly, keep a lookout for loose conclusions—where the authors make statements about significant findings that they do not actually test for (or over-reaching with their conclusions). Sometimes the discussion can also reveal methodological problems. Two prominent issues in the literature are drawing conclusions from tests that do not have a significant result and directly comparing the results of two different statistical tests. With hypothesis testing, remember that failure to reject the null hypothesis is not evidence for there being no effect. Similarly, finding an effect in one test and not in another does not mean that the two generative processes are different. Finally, it is often valid to temper the authors’ statements. For example, scientific papers should generally avoid value statements such as calling results ‘strong support’ for a given hypothesis or by interpreting a P value as being ‘highly significant’.
Novelty: Lastly, you may be asked, or choose, to comment on the novelty of the work you are reviewing. In general, a novel study helps progress by filling out a key gap, which can be achieved by using novel methodologies or by showing evidence for/against a classical hypothesis. Novelty is distinct from quality—the former can verge on an anecdote whereas the latter considers the robustness and depth of a study. As part of considering novelty, it can be very helpful for the editor to point out studies that are similar to the manuscript you are reviewing, or where it makes (or fails to make) an advance on previous published research. Here you could also focus on other published papers by the same authors to look out for cases of ‘salami-slicing’—or publishing very minor variants of the same work in order to boost an individual’s number of publications. Even if the review is double-blind, salami-slicing can be blindingly obvious.
Overall: Don’t forget when reading that your job as a reviewer is to highlight both the weaknesses and the strengths of the study. Also try and avoid bringing any biases into your review—even if you are not a fan of the hypothesis, don’t forget that scientists once rejected outright the notion that the earth is round.
Pitfalls to avoid when reviewing
Firstly, it is not your job to re-write the manuscript! For example, it is easy when you are still less experienced to focus on how the introduction is written. This should not be the focus of a peer review. Yes, the manuscript should be well-ordered and clear, but it still needs to represent the authors’ views and work. Remember, the reviewers’ job is to focus instead on scientific validity.
Another common mistake, especially with inexperienced reviewers, is to use what can appear as cues about the quality of work. It is really important that you don’t take the names of authors as evidence of quality. Many journals are now double-blind (so you’ll not know the authors’ names, nor will they know who reviewed their paper), but often you will have the full author list available. Big names in the field don’t always write great papers! Also – don’t believe everything you read in the acknowledgments (for example, acknowledging a stats guru doesn’t mean they followed that person’s advice).
Also remember that you are a qualified peer reviewer. You have been asked to review as an expert in the field, and therefore you should represent (at least) ‘the average’ audience for the paper. If you don’t understand something that you feel you should, this is not because you are a bad scientist. Instead, it is almost always because the manuscript lacks clarity. This in itself is a valid critique of the work.
Finally, don’t take the authors’ word for everything! For example, if the statistics don’t seem to agree with the statements in the text, then ask the authors to provide better evidence to support their statements. For example, make sure that strong evidence in support of a hypothesis is justified by large effect sizes rather than small P values (or worse still, neither). Lastly, make sure that the results actually support what the title of the manuscript claims to provides evidence for (it is remarkable that this is not always the case).
Asking for help
It is always advisable to immediately ask editors for help if you need it.
Extensions: Editors are almost always happy to oblige with giving more time to finish a review. Asking for an extension shows that you aren’t just ignoring your obligation.
Issues beyond your expertise: Sometimes you might not be confident enough to review some aspect of the manuscript, such as a particular statistical method. You might have accepted to perform the review, only to find out that the methods are highly complicated and beyond your knowledge. Here it’s best to immediately alert the editor. In general, they’ll be happy to receive your review (in which you can acknowledge areas that you cannot comment on). By telling the editor early on, this will give them the time to invite someone with greater expertise in that area to provide an additional review.
Getting help from a colleague: It is generally fine to seek help from colleagues, supervisors, or mentors. However, you should first check with the editor before sharing the paper (it is also important for the journal to keep a paper trail of who has seen the manuscript). In general, if there is an area of the manuscript that requires more specialist advice, and you know who can give that advice, you’ll be doing the editor a favour.
The process of reading for reviews can be very enlightening. However, it does take time. Having a strategy for how you will review the paper, and setting the appropriate amount of time aside to do this, will help minimise the costs of reviewing. Don’t forget that you will also need to write-up your review, so you should make sure to keep making notes as you read through the manuscript for the second or third time. In the third and final instalment of this series, we’ll provide guidelines on how to write up your thoughts as a finalized review.
If you want more information, the British Ecological Society has an excellent and in-depth guide that provides much more information than what we can here: https://www.britishecologicalsociety.org/wp-content/uploads/2018/12/BES-Peer-Review.pdf