Should authors decide whether their revised paper is re-reviewed??

29 January 2009 by Noah Gray, posted in Uncategorized

According to the Journal of Biology…Yes!! In an editorial released on Tuesday, new Chief Editor Miranda Robertson introduces us to a new policy at her journal, one she believes will begin to remedy the frustrating problem of iterative re-review prior to publication. By eliminating this process, the thought is that the journal will more appropriately re-claim its proper role; namely, as an entity of cutting edge research dissemination rather than one that “polices” the quality and accuracy of the research.

Sounds like a great deal…for the authors. But what about for the readers of the manuscripts who are used to believing that they have the comfort of peer-review protecting them as they apply their healthy skepticism when reading a manuscript? Doesn’t this blur the lines between unpublished and published work? And as for the editors, how will they make very difficult technical decisions in fields that are not in their immediate area of expertise without consulting an expert (i.e., reviewer)? 

If data are being published without peer review in journals claiming to be peer-reviewed, this is a problem. This policy seems to me like it will place J Biol somewhere between a peer-reviewed journal and a pre-print server. The reason I say that is because although the paper will have gone through one round of review, upon resubmission, which masochistic authors are actually going to check the box “please send this back to the reviewers, especially the one who wanted me to do lots of extra work”?? Rather, the authors are going to take this golden opportunity to convince the editors (and editorial board members) that the requests of the reviewers are “beyond the scope of the current work”, or that “these other controls provided in the revision are just as good to address the previous concerns”, or that the “new data we added greatly enhances the impact of this study”. In other words, a lot of data will be released for public consumption without having been scrutinized by a reviewer. Isn’t that what happens with a pre-print? At least with a pre-print server, one understands that the work may still be preliminary. But if one is not a regular reader of J Biol (or their editorials), won’t the typical assumption be that all data in the study was subject to refereeing?
There is a reason we review papers. It is because the editors and editorial board members are not experts in all disciplines and techniques. We identify 2-4 people that are highly-experienced in said disciplines and techniques, and allow those experts to come to some consensus on the value and accuracy of a study. This often leads to multiple rounds of review, with the authors adding new data to prove their points. Shouldn’t these new data be meticulously examined just like the data provided in the original version? Why wouldn’t authors just provide the safe experiments, and then toss in some more suspect, but perhaps provocative, results after the first round? If they do, would the editor feel comfortable making a decision on the paper without the assistance of a true expert? If the editors reject the paper, won’t the authors just go ahead and appeal that decision, perhaps even asking the editors to consult the reviewers in an effort to assuage the discomfort of the editor with the new data?
The arguments in favor of this policy, beyond the one given at the beginning of this post regarding the role of the journal in the scientific process, are weak. J Biol should have stuck to this high road instead of ticking off a list of debatable and more dubious supportive statements. The high road point?

…the policing function of journals (especially but not exclusively the high-profile journals) is in danger of overwhelming their primary function as publishers.

This, I think everyone can agree on. The way to fix this problem is where things diverge, and there is no quick-fix. As for the other support for adopting this policy?

  1. Authors can cite many examples of having a difficult time with a reviewer [Of course, but discuss these issues with the editor and sort out a compromise in each situation…that’s our job.]
  2. The content of the paper is the responsibility of the authors, not the reviewers [True, but then, why are journals frowned upon with every retraction? Why would a journal want to publish flawed studies, and then just point fingers at the authors? That’s a great way to lose not only submissions, but readership.]
  3. Seriously flawed papers still make it into the published literature after iterative review [True, but why adopt a policy that actually increases the likelihood that this will happen?]
  4. This policy will save the reviewers time [Yes, at the expense of losing those reviewers who disagree with a policy that marginalizes their efforts made and valuable time taken to improve a paper.]
  5. Sometimes reviewers dislike Nobel Prize-winning papers [Was that anecdote actually an argument for this policy?]
  6. Since journals still ultimately have an obligation to their readers, to hedge their bets, J Biol will include the concerns raised by the reviewers, and not necessarily addressed, in Commentaries that accompany all published works [No comment.]

This policy may save time in the dissemination of some papers; those only requiring textual changes or other minor revisions. But, many journals operate in this manner anyway when reviewers pass a paper and only require some t’s to be crossed. When this type of revision is submitted, it is thoroughly checked and sent to production straight away.
I don’t agree with this experimental policy, but will take a scientific mindset and wait for the data to roll in on how well it improves the dissemination of accurate knowledge.

7 Responses to “Should authors decide whether their revised paper is re-reviewed??”

  1. Maxine Clarke | Permalink

    I also blogged about this at Peer to Peer, though not with your admirable attention to detail.
    My main concern with this model of peer-review is that the referee reports are not intended to be posted with the incomplete ms. Rather, the journal plans to publish an accompanying commentary. If the commentary is not by the peer-reviewer or someone with access to the reports, the reader will not know what concerns were raised, specifically. If the commentary is intended to be by the peer-reviewer(s), why would they bother? Would they not prefer to have the author revise their ms in the light of their comments?
    Another question is that this journal does publish “accompanying commentaries” in any event, before introducing this new peer-review model, but sometimes they are not “accompanying” but “later”. Therefore, there is even more scope for confusion.

    Disclaimer: As an editor for another journal, my comments are not intended to be critical of J Biol in particular. They are rather a comment on this particular model of peer-review. That was also the spirit of my Peer to Peer post linked above.

  2. Irene Hames | Permalink

    As an editor for another journal, I’m also restricting my comments to the new model of peer review. I’m commenting from the standpoint of a professional editor who has overseen the handling of around 14,000 manuscripts over the past 18 years and written a book on peer review. (I’m posting both here and at Peer-to-Peer.)

    Noah and Maxine have made some very good points, and I agree with many of them. Noah’s post is a great summary of the issues and problems inherent in the new model, so I won’t go over what he’s already said. But this new model worries me greatly. When sweeping generalisations are made to promote a new model without providing evidence or references to back them up, that’s quite dangerous. Especially as outside of the main general science and highest impact journals, editors, and many editorial staff, come to their roles without any experience or training.

    One of my main concerns is that a crucial decision in the review process is being handed over to someone, the author, who is not impartial, but has a vested interest in a particular outcome. It’s the editor’s role to make that sort of decision, and to arbitrate between the various parties involved in the peer review process. If authors feel that reviewers are holding them to ransom, then the editor isn’t doing his or her job. It’s their role to assess what the reviewers say, taking into account many other pieces of information, and then advise the author on what needs to be done and what doesn’t, either because it’s totally unrealistic, or misguided. Peer review is a dialogue, and if an author doesn’t agree with what a reviewer says, or if a decision has been made on the basis of what the author perceives to be flawed information or reasoning, then they need to make a case and present it to the editor. A good editor is a reasonable editor. A dogmatic editor is a bad editor.

    It is very worrying that new data will be added without additional review if the authors decide they don’t want this, and Noah has explained why this is so. It’s quite alarming that the changes in such cases will be checked by the “editorial staff”. Will they be editors or assistants? Even if the former, there are many times when the judgement call can only be made by someone who is a specialist and, crucially, knows the current state of knowledge, problems and controversies in that field. In fast-moving areas it becomes even more important.

    It’s totally inappropriate for non-editor editorial assistants to check revisions. This happens. I have been contacted by concerned editorial assistants whose editors have asked them to do this and they’ve felt uncomfortable and out of their depth. I know from experience that the addition of a single sentence can have an enormous impact. Also, what about addition of things at inadequately monitored revision that will result in priority being inappropriately established? Or unwarranted claims of priority being made then?

    Part of the case made for the new model is to release more time for reviewers. But the new model is actually wasteful of reviewers’ time in one respect. If an author does opt to have their revised paper assessed by external reviewers again, it will go back to all the reviewers. But this shouldn’t automatically happen. It’s for the editor to step in and, based on the reviewers’ original comments, decide which of the reviewers needs to be consulted. The various reviewers may have been focussing on different aspects of the papers, they may have been asked for specific advice, some may therefore have been basically happy with the work except for minor issues. Others, may have picked up on more serious or complex points, and they’re the ones whose advice needs to be sought.

    So much hinges on the publication of work – grant funding, promotion, personal and professional prestige – that it’s crucial that all decisions to publish are made as fairly as possible and impartially. I can’t see how this is compatible with letting authors have such an important say at such a critical point in the peer review process.

  3. Maxine Clarke | Permalink

    Thanks for mentioning that you have commented at Peer to Peer, Irene, as I have just been to check and see your comment got caught in that blog’s over-enthusiastic spam filter. I’ve now made the comment live.

    I think the points you make are excellent ones. I have been a journal editor for many years, too, and I concur that the author’s perspective is far from impartial. I also know from our regular surveys of recently published authors, that they feel impartial but firmly applied peer-review has improved the final (published) ms.

    As mentioned, it will be interesting to see how this pans out – or what effect the policy has on the number of manuscripts published by this journal. A popular move with authors, I am sure.

  4. Irene Hames | Permalink

    Maxine, I sent a brief follow-up message to Peer-to-Peer with the link to Nothing’s Shocking and saying that Noah has blogged about the new model of peer review here. Can’t see it, so maybe it’s also got stuck in the spam filter. Without that info visitors to P-to-P won’t get the full picture.

    Sorry that I didn’t include it first time round, but I was dealing with a repeating ‘blue screen of death’ situation on my laptop. Can you make the message live please, or create a link in my original if you’d prefer (if messages are editable). Thanks!

  5. Cristian Bodo | Permalink

    Seems to me like the Journal of Biology is shooting itself in the foot by adopting this policy. If there is no guarantee whatsoever that the papers appearing on the journal have been peer-reviewed first, who’s going to send their manuscripts to them? (unless you’ve completely lost the hope that your manuscript can withstand the peer-review process, and are just trying to get it published no matter what). After all, even if your manuscript is flawless, just appearing there is going to immediately cast some suspicion in the rest of the scientific community, so the smartest move for prospective authors would be to send it somewhere else…

  6. Henry Gee | Permalink

    The proboscidean in this particular 3-space (oh do keep up at the back) is the situation of iterative peer review, and before I go any further I should declare that I am also an editor on the same journal as Noah and Maxine, which I joined when Miranda Robertson was also at the same journal. What an incestuous world it is. But I digress.

    One of the functions of an editor, in my view, is the responsible exercise of judgment. Even if there is an academic review board (a big difference between Nature and J. Biol.) it is the editor’s call to decide whether a particular manuscript could fly, at least in principle. For an editor to defer such decisions onto a referee is bad practice. The same applies when a manuscript is sent to review and the reports are in – an editor will do a lot of reading between the lines and weighing up arguments before making a decision, *and that decision is the editor’s.* It is not the referees’ decision, still less the authors’.

    In my experience, when one sees a manuscript trapped in a seemingly endless round of peer review, it is usually a sign that the editor has not been doing his or her job. Often, such a manuscript should have been rejected by the editor sooner, and has reached the situation because the editor has put off making a decision until the referees have made up his or her mind for him. More than a couple of rounds of iterative peer review are exhausting to authors, referees and editors alike. Referee-fatigue sets in, and if the final decision is rejection (which it probably should have been to begin with) weeks – months, even – have been wasted, during which time the author could have sent the rejected manuscript elsewhere and had it published.

    The buck has to stop somewhere, and that somewhere is the editor.

    Another misconception is that manuscripts have to be perfect. Miranda owns that this situation never happens, and it might not even be desirable, and this also implies risk on the part of an editor. Sometimes an editor has to exercise judgement as to whether a far-reaching paper is full of hot air (and should be rejected) or contains the germ of some genuinely new idea – in which case the editor should run with it and publish something which, on first look, seems speculative. But that decision must be one made by the editor, based on experience, advice, and, more than anything else, having an instinct for such things.

  7. Maxine Clarke | Permalink

    Irene – Your second comment (with link) is now live – was also caught in spam filter. If you don’t see it or the previous one, try refreshing. Hope that is OK now.
    Cristian – the papers are peer-reviewed once at J Biol. What the journal is now doing is allowing authors to publish them without making the changes required by the peer-reviewers (assuming that the editors – see Irene’s points – think the paper is basically sound).
    You may be right, but the J Biol course is, I imagine, going to be very tempting to many authors. Possibly intentionally so (on part of the journal). I checked out the comments to the editorial (linked here) earlier today to see what the journal readers think, but there are none (after I jumped through many hoops to get to the point where I could look!).


Comments are closed.