“…I can live with doubt and uncertainty and not knowing. I think it’s much more interesting to live not knowing than to have answers which might be wrong.“

Richard P Feynman, “The Pleasure of Finding Things Out”

 

It may be useful for non-specialists who are not actively involved in the scientific peer review process to get some insight into what constitutes “scientific peer review”. This blog post will give an overview of the peer review process, primarily based on my own scientific peer review experiences in biological and medical research.

The contemporary peer review process for manuscripts submitted to scientific journals usually involves the following stages:

1. Submission: The authors submit their scientific manuscript to a journal.

2. Assignment: Once the manuscript passes an initial quality control process, it is assigned to an editor, associate editor or academic editor with expertise that is at least broadly related to the focus of the manuscript.

3. Initial editorial decision: The assigned editor decides whether the manuscript is generally appropriate for the journal and category of article that it was submitted to. If the manuscript is deemed appropriate, it is sent out for review to experts in the field. If not, the manuscript is sent back to the authors and they are asked to submit it to a more appropriate journal. Occasionally, some editors may ask the authors to substantially revise the manuscript before it can even be deemed appropriate for peer review.

Many of the high profile journals, such as Nature, Cell or Science, use this stage to eliminate the bulk of submitted manuscripts. The criteria for rejecting manuscripts at this preliminary stage are rather vague and have more to do with the general goals of the journal (i.e. publishing high-impact papers that garner citations by other scientists and frequent mentions in the news media) than the scientific rigor and quality of the scientific work. Sometimes an informal nod to the editors from a renowned leader in the field via a brief email or phone call can also do the trick and help increase the likelihood of a manuscript being sent out for review.

High-profile journals also encourage a pre-submission inquiry in which authors can submit a brief description of the work instead of the complete scientific manuscript. This allows the editors to screen the potential submissions and only invite a select group of authors for a complete manuscript submission, if they are reasonably sure that the manuscript merits a full review. The precise percentages of manuscripts that are rejected at the initial state prior to the formal review or the comparative fate of manuscripts that are submitted after a pre-submission inquiry versus those submitted directly with an inquiry are very difficult to obtain.

When manuscripts are rejected prior to a review, authors usually do not receive any specific comments other than something along the lines of “your work is not appropriate for our readership” or “please consider submitting it to a subspecialty journal”. The peer review process would be far more transparent, if journals were obligated to provide accurate statistics on what percentage of manuscripts are sent out for review, what specific factors resulted in the decision on whether or not to formally review a manuscript and the acceptance rates for manuscripts that use a pre-submission inquiry versus those which do not.

4. Selection of peer reviewers: Once a manuscript has cleared the initial editorial hurdle, it is sent out for review by scientists who are chosen based on their areas of expertise. Most journals that I have either reviewed for usually require between two and four reviewers, although I have occasionally seen cases in which seven reviewers were asked to comment on a scientific manuscript. As science is becoming more interdisciplinary, scientific manuscripts increasingly span multiple areas of research and may thus require more reviewers with complementary areas of expertise.

Many journals allow the authors to suggest names for potential peer reviewers. It is generally understood that authors will not suggest active or recent collaborators as reviewers, but instead choose scientists who are most qualified. Authors are likely to suggest reviewers who will have a favorable opinion on the importance or significance of the research, but it is the editor’s decision on whether or not to accept the author suggestions or whether to select (additional) independent reviewers. A number of journals allow requests to exclude specific scientific reviewers in case the authors feel that these scientists may harbor strong biases against the submitted scientific work or against the authors.

5. Peer review reports: Once the experts accept the invitation to review a manuscript, they are usually granted 10-14 days to submit a completed report as to the significance, novelty and scientific rigor of the manuscript. This time frame is necessary because performing an in-depth peer review requires a substantial amount of time. Since peer review is conducted by scientific experts on a voluntary, unpaid basis outside of the normal work demands, peer reviewers often devote many hours of their evenings or week-ends to pour over the submitted manuscript, the figures, data tables and other supplementary data provided by the authors. A typical manuscript in the life sciences contains 4 to 8 multi-panel figures, each consisting of multiple graphs and other images. While short papers published in journals such as Nature or Science have strict limits on word counts and also restrict the number of figures in a paper to four, there are few limits on the supplemental data. I once reviewed a paper which had six multi-panel figures as part of the main manuscript, but also contained additional 10 multi-panel figures and three tables in an online “supplementary data” section!

A conscientious peer reviewer not only reads the submitted manuscript, but also tries to envision how the experiments were designed and conducted, and tries to put the findings into context of the existing literature. A six thousand word manuscript with six figures and a couple of tables represents a distillate of two or three years of research, often conducted by a team of scientists. The authors have had months or years to familiarize themselves with the experimental design, results and interpretation of the data, and some manuscripts are written in a manner that it is not easy to divine the actual intentions of the researchers. A well-written manuscript is much easier to review.

Most editors ask the reviewers to rank or grade the significance of the work, the novelty of the research as well as experimental design or approach. In addition to providing these grades or ranks, the reviewers also offer specific comments about the strengths and weaknesses of a paper. Many of the comments relate to the adequacy of the experimental design, the consistency of the results, whether the interpretations match the presented data and how these new findings relate to previously published work. These comments vary substantially from reviewer to reviewer. Some just write a handful of sentences, others write two pages of comments.

Finally, reviewers provide the editors with a confidential overall recommendation, such as reject the manuscript, return the manuscript to authors for major revisions, return manuscript to authors for minor revisions or accept the manuscript as is.

The opinions of reviewers can vary substantially, because reviewers differ in terms of their priorities for what constitutes significant research, their analytical skills, their personal biases and their threshold for what is acceptable for publication. It is not uncommon to have one reviewer outright reject a manuscript (because the manuscript would not improve much with revision) whereas another reviewer wants to accept a manuscript pending minor revisions.

 

6. Editorial decision: The editor receives the reviewer reports and has to decide upon the overall verdict on the manuscript. If the reviews are too disparate, the editor may solicit the opinion of additional peer reviewers to help with a final decision. The final decision letter contains editorial comments as well as the comments provided by the reviewers to explain why a certain decision was reached. Many journals send a copy of the decision letter to the reviewers, not only to maintain transparency in the decision making process but also to allow peer reviewers to read the reports or comments of the anonymous reviewers. This is a form of feedback and learning opportunity, which enables peer reviewers to assess whether their fellow reviewers picked up on strengths or weaknesses of the manuscript which they may have missed. If the decision is made to ask the authors for revisions, the authors are usually given a time frame during which to make the revisions and the revised manuscript then again undergoes the review process. The editor may decide to use the same reviewers or choose additional peer reviewers.

 

******

I chose to elucidate the peer review process in such detail because I want to highlight what “peer review” does and does not entail. At its core, it is just a review of the submitted data, not a validation of the data. Peer reviewers do not perform any experiments to check the accuracy or replicability of the data contained in the submitted manuscript. The assessment of the validity of the results does not occur during the peer review process, but months or years later when other scientists attempt to replicate the published paper.

In most cases, peer reviewers do not even have access to the raw data. It is therefore very difficult for a peer reviewer to discern whether or not scientists have chosen to submit truly representative data, or whether they chose the “best” data which optimally supports the conclusions of the manuscript. Much of scientific peer review is based on the honor system. If researchers claim that they have performed an experiment five times or that the results are statistically significant, the peer reviewers take the word of the researchers and base the review of the scientific results on this assumption.

Peer-reviewed research is understandably more rigorous than research which has not undergone any review process, but the review process is quite limited in its scope and prone to errors due to the subjective priorities and biases of editors and reviewers. Validation of the research occurs when independent scientists are able to replicate the published findings. Replication of scientific results is the gulf which separates peer review from peer validation.