Reviewing papers part III: Writing the review

In parts I and II of this series, I discussed how to get involved in peer review and how to critically read a manuscript. The final step is to communicate your opinions about the manuscript to the editor and to the authors. There are three parts to this: (i) your recommendation about what steps the editor should take next with the manuscript, (ii) confidential comments to the editor, and (iii) formally writing a review that will be seen by the editor and the authors. In this third installment of our series on how to review papers, I discuss how to make a recommendation and how to structure the written review. 

Written by Damien
@Damien Farine

The recommendation
When you log into the peer reviewing system to submit your recommendation, the first thing you will be asked for is to fill out one or more questions about the manuscript. In all cases, you will be asked to make a recommendation, which ranges from reject to accept. I have always found this step to be tricky and, in my opinion, it should not be the reviewer’s job to make this recommendation (some journals now allow you skip making a recommendation). What also makes it tricky is that the decisions by journals often don’t align with the list of recommendations; specifically, journals will ask whether to choose reject versus major revisions, but in most cases now journals use the option ‘reject with the possibility of resubmission’ instead of major revisions. In general, as a reviewer you will be given 4 choices:
 
Reject: The paper has major flaws and is clearly unsuitable for publication, at least in the current journal or current form. Here the recommendation is that the editor declines any further options for the authors to publish their work in that journal. This sounds drastic, but keep in minds that it is common for well over 50% of all submissions to be rejected. I use a rule of thumb that a paper should be rejected if the changes needed could result in new findings or changes to the conclusions, meaning that it could become an entirely new paper (which could include testing an alternative hypothesis or rejecting rather than accepting the current hypothesis).
 
Major revisions/Reject & resubmit: The paper could be suitable for publication, but would need some major work done to it. Doing so would change many aspects of the results, but would be unlikely to change the overall findings—i.e. whether the study supports or rejects a given hypothesis. Changes are often to the analytical approaches, but can include changes to the framing or emphasis of the manuscript.
 
Minor revisions: Small changes are needed, usually to the presentation of the work, almost exclusively to increase the clarity. This can include how the introduction is written, how graphs are presented, whether all the necessary statistics are included in the results, etc. However, publishing the manuscript as it is would not be ‘wrong’.
 
Accept: The manuscript could be published as it is.
 
Other questions: Some journals will ask additional questions, such as how robust the study is, whether data are clearly made available, if the study is novel, whether the manuscript is suitable for the journal, and whether you are confident in reviewing all aspects of the work.
 
Confidential comments to the editor
The first part of the written review consists of comments to the editor. This section is optional to fill, and is usually much less formal than the comments to the authors. Some reviewers write a sort of ‘cover letter’ to the editor highlighting their key points. Others state their opinion much more bluntly than they should in the comments to the authors. In my view, this section is where you should voice any opinion you have about the novelty of the research. This is very useful for the editor—you are after all an expert in the field—but writing your opinion about novelty in the comments to the authors can unnecessarily tie the editor’s hands when it comes to making a decision based on multiple reviews. In this section you can also write about any concerns that you think might be worth suggest chasing up (e.g. by asking a more specialised reviewer), or highlight to the editor if you do not have sufficient expertise to critically evaluate a particular section of the manuscript. Your concerns can include ethical questions—if you are not convinced about the ethics for some aspects of the work, do write these down so that the editor can chase it up (such as by inviting a specialist ethical reviewer).
 
Overview of the structure of the comments to the authors
The main part of the written review is the comments to the authors. These have evolved to have a fairly common (but not compulsory) structure. The structure consists of three parts: (i) the summary paragraph, (ii) the major conceptual or methodological comments, and (iii) the minor or detailed comments.  The bulk of the review (say about 70% of the work the authors should need to do) will come from the major comments.
 
The summary paragraph
The first paragraph should be a summary highlighting the questions and/or hypotheses, the key findings, and how the conclusions were drawn (i.e. the methods). The aim of this summary is ultimately to demonstrate to the authors that you understood the paper, or how you understood the work that they presented. This summary paragraph should focus on the positives of the work—what it is that the manuscript actually achieves. If possible, try to put this in perspective of the broader literature.
 
The major comments
The major comments represent the main points that you believe the editor and the authors should seriously consider before the work can be published. These are generally over-arching comments that transcend the whole paper—addressing one of these comments could force the authors to re-write significant sections of the paper, that could include revising at least some of the message of the paper. The aim here is to make clear arguments for or (usually) against a given section of the manuscript. However, this requires much more than simply stating that what the authors did is wrong, but to make it clear how and why you came to your conclusions, and, if possible, giving as much advice to the authors about how to overcome the problem. Areas of a manuscript that are often the subject of major comments include:
 
The philosophy of the paper: is the logical deduction of the prediction(s) valid? We rarely directly test a hypothesis, but instead argue that if the hypothesis is true, then some biological relationships should also be true, meaning that finding evidence in support of these would result in support for that hypothesis. For example, if living in a larger group provides safety from predators (the hypothesis), then we expect individuals in large groups to be less vigilant than those in smaller groups (the prediction). However, being in a larger group also entails greater within-group competition, which could also increase vigilance, thus introducing a confound that could cause the original deductive reasoning to be invalid.
 
General methods: given the predictions are valid, are the data collected and tested correctly? Or, are they even actually tested? It is remarkable how often manuscripts do not actually test the question they set up. Remember, the statistical model represents the actual question being tested, irrespective of what the authors claim to be asking. Another important aspect of the general methods to focus on is the data collection methods. Were the field methods and sampling design appropriate? Also critical is sample size. There is now very strong evidence that small sample sizes are much more prone to false positive findings. Do the authors consider this possibility? In reading the manuscript, reviewers can often find hints that authors have explored their data extensively before coming to the final model, which could massively increase the chances that the results presented have no truth. An example of this is when authors include a very specific set of covariates in a model.
 
Statistical flaws: any concerns about the validity of the statistical methods, including asking for more information to be provided, represents a major comment. Common comments here include questions about the choice of statistical model (or family of models), whether authors have dealt with auto-correlation or any other source of non-independence, or the choice of covariates to fit into a model. Authors also often leave out valuable information from their methods and results (usually by mistake), including details of the model itself or tables containing full statistical outputs.
 
Fit to the literature: does the manuscript properly acknowledge existing works? It is critical that the questions and findings of a manuscript are properly embedded in the existing literature. In particular, a paper should give an honest appraisal of alternative perspectives. Yes, it is fine to argue for a particular hypothesis, but this does not justify excluding alternatives. One thing though: reviewers can often be very judgemental when authors have missed a key citation (or even sometimes a whole body of work), but keep in mind that this could be an honest oversight.
 
Alternative explanations: a critical part of being a reviewer is to question whether there are alternative explanations that could explain what they authors claim. For example, early naturalists who observed starling flocks claimed that only telepathy could explain the striking collective behaviours that they perform. Yet we now know that these can emerge from very simple individual-level behaviours. If there is a remote possibility that an alternative mechanism could generate the same patterns that the authors use as evidence for a given hypothesis, do not hesitate to push them to explicitly exclude it.
 
Minor comments
Minor comments are often called ‘line by line’ comments, because these are usually much more specific to a particular segment of the manuscript. Usually this is a list of comments, where each comment is preceded by the relevant line number, thus pointing the authors directly to where a change is (possibly) needed. These comments can include asking for extra information, asking for clarification (e.g. re-writing a sentence), typos, or even suggesting additional relevant citations. It is helpful here to also highlight which sections made you think about some of the points in the major comments. The list of minor comments can be quite long or quite short. If a paper is good, you might want to invest a bit of time giving tips on where authors can improve the clarity of the writing. If the methods are flawed, then highlighting typos will not be of so much use.
 
Writing style
Finally, it is very important that the review process remains constructive. This means that a good review should be written almost as a helpful guide rather than a critique. In general, the whole tone of the review can hinge on just a few words here or there. First, be sure to give positive comments or suggestions, and to highlight good parts of the work and not just the bad parts. Second, be careful with the choice of words. Be polite! Avoid using direct or accusatory wording, such as ‘This is wrong’, or even directly addressing the authors, such as “You did not consider …”. It is best to use indirect language, such as “The authors may need to consider the alternative that …”.  Ultimately, it is best to write the review how you would like someone to critique your own work.