Impact, like a drop in the ocean.

19 April 2012 by Pete Etchells, posted in Uncategorized

Impact Event

I really disliked exams, and I’m really glad I don’t have to do another one (well, as far as I know). I used to get really nervous, even in the weeks before when I was revising, and I seemed to have blanked the actual experience of sitting in an exam hall from my memory. I think the holy grail for students of all ages, especially at this time of year, is an easy and innocuous quick fix to increase your exam performance. Well, according a news release on Wednesday, the answer has been with us all this time. “”">Bring water into exams to improve your grade", proclaims research presented at the British Psychological Society Annual Conference this week.

The observational study looked at 447 students across a number of years taking their exams at the University of East London. The results suggested that around a quarter of the students took bottles of water into exams with them, and after taking into account academic ability, these students showed an improvement of around 5% in their exam grades. While the researchers admitted that they didn’t know precisely why this might have occurred, suggestions included “theory is that information flows more freely between brain cells when they are well hydrated” and that “water consumption may also alleviate anxiety, which is known to have a negative effect on exam performance”.

Compelling stuff, but there are a few things that bother me about this report. For one, I think the PR machine has jumped the gun this time – the results haven’t been published in a peer-reviewed journal yet, they’re just from a conference presentation. As such, it isn’t clear what precisely was controlled for. Given that the study was observational, the answer to that question is probably ‘not much’. Which means that any speculation about psychopathological and physiological interactions and effects is a bit pointless really; we don’t know that such effects explain the results any more than saying that people who are generally more prepared for exams will probably be more likely to bring water with them. It’s also worth pointing out that nowhere in any of the press releases does it say that the students actually drank the water. Just that they brought a bottle in with them. Hmmm.

Perhaps I’m being unfair – it’s quite clearly a PR gaff and they should have waited until the full study was published. Normally, I would have let it slip, but something in the final paragraphs in the BBC article raised my hackles. Not the old “Future research is needed to tease apart these explanations” gambit (really? I thought we’d done with all that testing and science malarky now). Nope, it was the point that “There are also implications for policy makers in terms of the availability of water on campuses”. Impact statement: tick.

Now, don’t get me wrong – I’m all for thinking about how your research might actually contribute something meaningful to a population broader than your scientific readership. But aren’t things getting a bit ridiculous now? Do we have to justify absolutely every piece of research in terms of immediate and direct impact on policy? What does that even mean any more? We’ve got a case in point right here – what are the implications for policy makers, exactly (and who are these policy makers)? Making water more widely available on campus doesn’t necessarily mean that students are going to bring more of it into exams. Or do we need to start forcing water bottles into their hands as they walking in to the exam room? Perhaps a little water fountain at each desk, or an intravenous drip?

I’m probably being a bit over-grumpy about it all; I just worry that in the drive to tick more boxes to get funding, scientists are having to come up with meaningless statements to justify their research in a wider context, and really, that doesn’t benefit anybody.

5 Responses to “Impact, like a drop in the ocean.”

  1. Suzi Gage Reply | Permalink

    Maybe those people who brought in water had written the answers on the inside of the bottle labels?! Did they check for THAT?!

  2. Pete Etchells Reply | Permalink

    Oh, snap. Hadn’t thought of that one…

  3. Mark Gardner Reply | Permalink

    Yes, perhaps a little over-grumpy.

    Your blog is an engaging piece of writing. Your point about the value of being skeptical until results are published in a peer review journal is well made. Skeptical is good.

    However, I believe it best to try not to be cynical about research or about the intentions of others. Especially, when based upon scant information. I agree with you that the evaluation of the credibility of this research is best deferred to the peer-reviewers. Or, at the very least, to those that heard the full presentation of results at the conference.

    I do, however, object to your claim that the authors came up with “meaningless statements to justify their research in a wider context”, and that this was motivated “to tick more boxes to get funding”. In my view, the value of the research comes from it being situated in an authentic real-world context. The study may only shows an association between fluid availability and examination performance. But given that it converges with the findings from controlled laboratory experiments (that have been published in peer reviewed journals) the wider significance seems clear. I have experienced students complaining of the difficulties getting a drink on a metropolitan university campus, where the historic buildings have been poorly adapted to provide drinking water, and commercial bottled waters are expensive.

    If you’re ever in London, I’ll offer to buy you a beer (or a bottle of water) and talk you through our results.

    (I need to disclose that I am a co-author on the paper)

  4. Pete Etchells Reply | Permalink

    Hi Mark,

    Thanks for the comments, good to hear from you! I think I probably didn’t explain myself fully with regards to the impact statement stuff. I wasn’t having a go at your research in particular; I was using it as an example of what I feel is a wider issue about the funding bodies requiring justification of research by including statements on impact. While this in and of itself isn’t a massive problem, I worry that ‘impact’ is increasingly becoming more and more of a major focus for research funding, and if your work isn’t obviously applied in nature, you struggle to get money.

    I used your press release as an example, well, mainly because it’s been picked up by the national press. But also, because the point that the piece ended on about informing policy just seemed really vague, almost tacked on at the end as a soundbite, and (in my opinion) spoke to the worries I’ve just outlined.

    I’d love to hear more detail on what you think would help; you mention in your comment about your own experiences with students having difficulties accessing water, but was that factored into the study? Also, I think my point still stands about general preparedness for an exam being an important factor that wasn’t mentioned in the pieces that I read.

    I apologise if my post seemed unduly harsh. I was simply going off what I had found in BPS press releases and the national media. But please bear in mind that most of the people who read those articles will similarly have not much more information to go off, and reading through some of the comments in the press, a lot of people ultimately seemed vexed that your research – as it was presented – had some big holes in it. I’m sure the research is solid, but the press release was pretty wishy-washy.

    There’s been a lot of talk in the media recently about how scientists interact with journalists and how to improve such matters, and I’d be happy to send you links to relevant articles if you’re interested.

    And I’ll definitely take you up on the drink next time I’m in London :)


  5. Jenna Todd Jones Reply | Permalink

    I had considered that it’s generally better advice to drink water in small doses consistently (if that is what the students are doing) rather than in large amounts all at once for good hydration, and also that this was hopefully based on some kernel of empirical truth – it’s interesting to see that this might contribute to brain fitness as well as physical fitness.

    Also, since when did 5% changes cross grade boundaries? :-| I’ve been out of the exam game too long…

Leave a Reply

× 1 = eight