The big pre-registration debate

25 July 2013 by Pete Etchells, posted in Uncategorized

At the start of June, Dr Chris Chambers and Prof Marcus Munafo, along with over 80 other signatories, posted an open letter in the Guardian calling for more journals to role out a new type of research paper, in which methods and analyses are peer reviewed before data are collected. Chris has blogged about this initiative before - see here. Today, Prof Sophie Scott published an article in the THE, in response to that original Guardian letter, criticising pre-registration. I think it's great that we're having a debate about this, because it means more people are thinking about it. Moreover, I'd be wary of *any* system that all scientists blindly adopt without question. To try and facilitate the conversation, I've set up a storify which attempts to follow the debate on twitter (see below). If you want to join in, use the hashtag #prereg, and I'll update as best I can. I've also set up a Branch conversation for those who don't like the 140 character limitation of twitter - you can find that by following the link below (also appears embedded at the bottom of this post). Lastly, if you want to raise questions or points here, feel free to do so in the comments section. I'll also update this post with any and all relevant links to blog posts and other material. If you have any links you'd like me to add, drop me a comment.

FAQs about Pre-registration, by Chris Chambers (jump to end of the pdf)
Storify of the debate on Twitter
Branch Conversation
Neuroskeptic
James Kilner | (follow-up)
Daniel Lakens
Michael Collins
Ase Kvist Innes-Ker
Suzi Gage
Dorothy Bishop July 2013 | Jan 2013
Nate Kornell
Martin Larsson
Aidan Horner

 


5 Responses to “The big pre-registration debate”

  1. Oasis Reply | Permalink

    Thanks for this nice overview!

    I would like to add that I always think "debate" is only useful when people actually use decent argumentation and try and investigate whether their own view could be flawed. Randomly saying things to "show an alternative view" is pretty lame i.m.o., I also don't think it's scientific.

    Interesting question perhaps (e.g. when looking at the twitter overview): did we have a debate or were people just throwing opinions at eachother? were possibly flawed arguments exposed ? was there agreed concensus on some points at least, and did these really matter with regard to the primary disagreement? was there agreed concensus that clarified on what specific points people disagreed (so at least it would be clear for now what the issue is)?

  2. B.R. Reply | Permalink

    So, when reading the piece in THE I am trying to find arguments described against pre-registration. Here are some (if I did this correctly):

    1. "They claim that pre-registration, which would involve journals accepting future papers based on the design of experiments rather than their results, would greatly reduce such questionable practices since the incentive to indulge in them to make papers more publishable would be substantially reduced. However, there are numerous problems with the idea. Limiting more speculative aspects of data interpretation risks making papers more one-dimensional in perspective."

    To me the important thing seems to be that a) it is claimed that there are "numerous problems with the idea" and b) the "idea" refers to "They claim that pre-registration, which would involve journals accepting future papers based on the design of experiments rather than their results, would greatly reduce such questionable practices since the incentive to indulge in them to make papers more publishable would be substantially reduced"

    Now, when I take a step back and try and look at the logic behind pre-registrating, it seems to me that QRP's would be reduced, because you could not do (some) of them (e.g. only reporting 2 out of 8 analyses). Regardless of that, it seems to me that the next line "Limiting more speculative aspects of data interpretation risks making papers more one-dimensional in perspective" has nothing to do with "would greatly reduce such questionable practices since the incentive to indulge in them to make papers more publishable would be substantially reduced". I don't see the logical connection and/or validity of argumentation, but I could be wrong.

    Now, aside from this and just looking at the arguments against pre-registration I would like to follow a more general perspective: do these arguments matter in the discussion pre-registration system vs. non pre-registration system.

    2. "Limiting more speculative aspects of data interpretation risks making papers more one-dimensional in perspective."

    You could still speculate, only you would show that you did.

    3. "And the commitment to publish with the journal concerned would curtail researchers’ freedom to choose the most appropriate forum for their work after they have considered the results."

    True in the strictest sense (when you pre-register with a journal), but why is this a "bad thing"? I gather researchers can live with the fact that their results are being published in that journal, why else would they have opted to pre-register there? Regardless of that, I think researchers who would have problems with this can also pre-register their study on their own and look at the results and then send it to a journal of their choosing. Sweet!

    4. "With no results to go on, reviewers would be more likely than ever to rely on reputation, which would count against junior scientists. "

    I gather pre-registration would also be anonymous, just like sending stuff to a journal without a pre-registration option.

    5. "In addition, the requirement to refine studies and their interpretation prior to data collection would prevent us from learning from our mistakes along the way."

    That's why you have pilot-studies. And wouldn't that also happen without pre-registration? I mean, at a certain point you start collecting data and it seems not a good idea to change things after that point.

    6. "Moreover, in my fields (cognitive neuroscience and psychology), a significant proportion of studies would simply be impossible to run on a pre-registration model because many are not designed simply to test hypotheses"

    I gather the pre-registration call was not meant for those kinds of studies. It's like saying "your p < .05 rule of significance is not good, because I am not using statistical analyses but am writing a theoretical piece". I think it's kind of implied that the pre-registration call is meant for certain kinds of studies, e.g. those with the possible use of QRP.

    7. "I am also very uncomfortable with the model’s implication that hypothesis testing is the only correct way of doing science. This may be true for the clinical trials from which pre-registration takes its inspiration, but we have known since Thomas Kuhn that scientists don’t just proceed, study by study, testing individual hypotheses."

    see comment under 6.

    Now, the thing is that when I look at these arguments and the text as a whole, the arguments hold little ground given the premise of the reason for pre-registration [e.g. "Drawing on a paper that asked psychologists to self-report their own dubious behaviour, they argue that large numbers of life scientists cherry-pick data, hide null results, fail to employ adequate statistical power and reinvent the aims of studies after they have been completed to make it look as though unexpected findings were predicted.

    They claim that pre-registration, which would involve journals accepting future papers based on the design of experiments rather than their results, would greatly reduce such questionable practices since the incentive to indulge in them to make papers more publishable would be substantially reduced."]

    When I look at the entire piece, I wonder if the author mistook the message of the pre-registration call and overlooked the possibility that this call is only meant for certain kinds of studies. Within those kinds of studies, pre-registration would/could probably solve some issues regarding QRP's.

  3. Monado Reply | Permalink

    I think you mean "roll out".

  4. Michael Carroll Reply | Permalink

    Thanks so much for the link to my blog and for taking this issue on. Please note that my last name is "Carroll" rather than "Collins" (not that there is anything wrong with "Collins" per se).

Leave a Reply