The pseudoscience of anecdotes

6 June 2012 by Pete Etchells, posted in Uncategorized

The results from the TREAD study (TREAtment of Depression with physical activity) came out today in the British Medical Journal – you can check it out here. TREAD was a long term randomised controlled trial (considered the ‘gold standard’ in clinical testing) which aimed to figure out whether giving people who were suffering from depression a bit of help with their exercise regime (in other words, having a facilitator provide advice and encouragement to the patient) actually improved their mood or not over time.

It would be great if it did – it would mean that we’d have an alternative to drug-based therapies that would be simple for doctors to prescribe, and desirable for patients who were worried about the side effects of antidepressants. The trouble with the results from TREAD, though, is that they didn’t show any benefits for those who received the intervention, when compared to those who didn’t – either in terms of improved mood or reduced antidepressant use. In fact, the only thing that did improve was the intervention group’s amount of physical activity during the follow-up period.

If his article in the Guardian today is anything to go by, this seems to be a huge irritation for Simon Hattenstone. The article, in fact, provides a masterclass in what you should look out for in a poor piece of science journalism. Let’s have a look what the problems are.

“So all of you who thought you were learning how to cope with your out-of-kilter brain, who had worked so hard to release endorphins and get a serotonin surge, who had made life manageable by running, going to the gym, dancing, or whatever, were WRONG.”

Making out that the the study is telling people who are depressed that if they’ve been exercising, it’s a waste of time, is a good way to get readers on side. It tends to rile people up, and make them amenable to the point that the writer wants to make.

“According to this report, carried out by the universities of Bristol and Exeter, and funded by the Department of Health, you exercise-tastic depression-battlers are simply deluding yourself. The study is, apparently, the first large-scale, randomised controlled trial to establish whether exercise should be used in primary health care to help treat adults with depression.”

Or, y’know, just spell it out in no uncertain terms – ‘this report is telling you that you’re stupid if you’re trying to make yourself feel better with exercise.’ It’s got good shock value. Also, note the use of the word ‘apparently’ in the second sentence – it makes it sound like the research is trying to make itself sound more important than it actually is. For reference, the actual words from the BMJ article are ‘the trial is one of the largest trials of physical activity and depression to have taken place in primary care.’


“As a depressed manic exerciser, who has found running hugely helpful, I would like to blow a great big Panglossian fart in the face of this churlish research.

Writers who have a drum to beat tend to identify themselves as authorities-from-experience on some subject or another. It lends weight to any points they make because they can say “I’ve been through that, and I know that X actually does work for me”. And actually, that’s a perfectly valid opinion to have – I don’t wish to belittle anyone in this way at all. The trouble is, it’s not particularly scientific. For example, I’ve suffered from moderate-to-severe depression over the past fourteen years, and I’ve often taken to the gym in the hopes that it would help my mood. It’s never worked. Actually, at times, it’s made me feel worse about myself, because I could never stick to it. Does that mean that exercise doesn’t work for depressed individuals? Not at all, it just meant that it didn’t work for me. It’s in this way that arguments from experience are extremely compelling; the ones it works the best on are those that make the arguments in the first place. As we’ll see later on, it’s also why they’re potentially the most dangerous sort of evidence.

First of all, slightly less depressed, even if not statistically valid, has to be better than the same or more depressed."

And here we get to the crux of the problem. The results from TREAD found that those in the intervention group were very slightly less depressed at the end of it all, when compared to the control group, but this difference wasn’t statistically significant (or, in the article, ‘valid’). ‘Significant’ (and ‘valid’, but for different reasons) is one of those words that means different things for scientists and the public. In everyday terms, if something is significant, it means that it’s really important – for example, the European Championships is a significant tournament in the footballing calendar; it’s a huge honour if you even get picked to play in it, let alone win it. So when you say that something’s not significant, like a result from a study in which some people become slightly less depressed, the impression you get is that of a cold and heartless scientist; someone who doesn’t care that a patient got a bit better, because it ’didn’t fit with the rest of the results’.

Significance in the scientific or statistical sense, though, doesn’t mean this. If something is statistically significant, it means that it is unlikely to have occurred because of random, chance factors. What that means for the TREAD study is that, yes, some people did get a bit better, but we can’t be sure whether that’s because of the intervention they were given, or because of something else that happened to them. In this way, it’s a form of caution. It’s great that some people got better – it would be great if even more did – but if they didn’t get better because of the treatment, then there’s no point in rolling that treatment out (at great expense) across the NHS.

"Second, it is unclear, to me at least, what stage of depression the participants were at. Yes, if somebody was so depressed that they can’t face getting out of bed, ordering them to go for a 10km run probably wouldn’t do the trick. Indeed, it might make them feel considerably worse.

This is actually a fair point – it’s really good practice to critique studies in this sort of way. We have to be sure that the people being tested are appropriate for the study. Moreover, it’s difficult – as the article states – to compare how the intervention might affect someone who is diagnosed with dysthymia to someone with severe depression. You need to try and keep your patient population as similar as possible to each other in lots of different ways, so you can be more sure that the results you get are because of the intervention you’re providing, rather than because of individual differences in the patients. From reading the BMJ article, it looks like the researchers performed a pretty comprehensive baseline screening session, and had quite detailed exclusion criteria. That doesn’t mean that it’s perfect; scientific studies never are. But the right sorts of protocols seem to have been followed.

However, we depressives don’t “use” exercise like this – it’s not a panacea, it’s a means of managing depression once we’re beginning to feel better or when it is in remission; a way of keeping it at bay."

So just when you think the article’s getting better, your hopes get completely undermined by this sort of statement; lumping ‘depressives’ together into a characterless group, defined only by the fact that they’re all depressed. Making a point about how it’s important to consider individual differences, and then completely ignoring your own advice straight away is an automatic fail and renders the rest of the argument redundant.

I’ll finish by going back to the points about arguments from experience, summed up nicely in the final sentence of the article:

Perhaps we should rely on self-knowledge rather than research when it comes to depression. After all, nobody knows your own body and mind quite like you do. So sod the academics, I’m off for a run.

No. No no no no no. No. The absolute last thing that we should be doing is relying on self-knowledge. If you rely on anecdotal evidence, then health policies (any sorts of policies, actually) just become a shouting match where whoever yells the loudest gets their way. Sometimes, when I get depressed, I eat loads of chocolate. Sometimes it makes me feel better. If we were to appeal to self-knowledge, then on the basis of my ‘evidence’, the NHS would be sponsored by Cadbury’s and we’d be paying £7.40 for a bar of Dairy Milk at the pharmacy. And weirdly, it might actually work for some people. The thing is, other treatments would work better and for more people. So for the vast majority of people, eating chocolate would not only be a monumental waste of time and money for everyone involved, it would also have the more threatening implication that people would not be having the proper treatment that they should be getting – the one that actually works. The only way that we can figure out what those treatments are is through randomised controlled trials and solid, objective research.

So, what we need to take home from the TREAD study is that, while having someone support you and provide advice on your exercise regime might help some people with depression, it doesn’t seem to work for most – so at this point in time, it’s probably a good idea not to spend loads of money rolling it out across the country. We need to look at it again, and think about why it’s not working (and who knows? That sort of research might find something that does work for everyone). What we shouldn’t be doing is writing sensationalist articles that misrepresent both the study in particular, and science in general. It doesn’t do anyone any favours.


8 Responses to “The pseudoscience of anecdotes”

  1. Neuro Bonkers Reply | Permalink

    My problem with how this research has been presented is that only 15% more people in the exercise group actually exercised than in the control group. The study therefore tells us nothing about how exercise affects depression, it instead answers a different question entirely – when people with depression are asked to exercise, do they actually exercise? This study simply shows us that with the TREAD intervention, the answer is no.

    In my opinion the comment piece in the Guardian, which as you rightly say is full of flaws is actually less flawed than the Guardian’s news segment on the research which is titled “Exercise doesn’t help depression, study concludes”, which is simply not an accurate portrayal.

    My blog post on the coverage: http://neurobonkers.com/2012/06/06/breaking-news-simply-telling-depressed-people-to-jog-on-does-not-help-relieve-symptoms-of-depression/

  2. Katherine Brown Reply | Permalink

    I think if this had been a news report, then ripping it apart in this way would definitely be sensible. But Hattenstone’s piece was on the Comment Is Free blog, and was an opinion story. People with depression do tend to feel very strongly about what is working for them, and what isn’t, precisely because the balance between the two is so incredibly perilous. For example, anti-depressants worked for me once I found a good one, but exercise didn’t simply because most of the time I either couldn’t get out of bed, or hated myself too much to allow myself to try again.

  3. Baking Biologist Reply | Permalink

    I agree that Hattenstone’s piece was a bit over-emotional, and missed a few things (e.g. the BMJ article does explain that people had to have a BDI score of at least 14, which – if you know depression – does mean something), but I think it was inevitable based on the headlines.

    What Chalder et al have shown is that a particular exercise initiative didn’t work for a very broad range of people. They didn’t block the study by sex or by age or by previous activity levels. They can’t say whether, for instance, young women benefited more from the trial than older men. They can’t say that in participants who could run 5k the endorphins were helpful, but going for a short walk didn’t have the same impact.

    That’s the problem. The confidence intervals for their data are HUGE because there was such variation in the population (which could easily be caused by the fact that only 15% of the exercise group met their exercise target! – I’m assuming that the number of ‘non-exercisers’ managing 2.5 h a week of moderate exercise is zero). But instead of reporting this as ‘there could be an effect, but on average it’s hard to see one’ this is reported as EXERCISE IS USELESS FOR DEPRESSION. And for once I don’t think lazy science journalism can be blamed for the over-reaching headlines.

  4. Pete Etchells Reply | Permalink

    All interesting points, and thanks for posting them. I agree that there are a number of issues along the chain with this story – Martin Robbins at the Guardian has explained this pretty well today here ( http://www.guardian.co.uk/science/the-lay-scientist/2012/jun/08/1?CMP=twt_gu ). There seems to be a disconnect between what the paper is actually reporting, and what the press release talks about. This in turn gets filtered down through the media incorrectly.

    And fair enough, that Hattenstone’s piece is an opinion story, but why should that absolve it of responsibility to represent things appropriately? The piece seems to be completely disinterested in what the paper actually said, instead aiming to assemble a load of single-point anecdotal stories to back up the argument that this research is worthless, ‘because it worked for me’. The trouble with that, is if you get 10 or so people who all say the same thing, it starts to sound convincing, but it’s not a particularly useful way to find things out about the world, for reasons I outline in the above piece. And it puts researchers in a really bad light, which doesn’t help anyone.

    The research isn’t perfect, but research never is. It’s an incremental step in our understanding of what does and doesn’t affect depression. The results are the results, and we need to use them objectively to inform further research. I don’t think anyone’s going to be insulted if the research is critiqued in a constructive way, because it means that someone can do another study that’s an improvement, which, in the long-run, makes things better for patients. Throwing your toys out of the pram and telling academics to ‘sod off’ has the opposite effect.

    Also, I have a real issue with this whole ‘we depressives’ thing that both Hattenstone and Rice-Oxley go on about in their respective pieces. It’s not a club, and I’m sure there are lots of people who don’t appreciate being defined by their illness, whatever it may be, by a self-appointed spokesperson.

  5. Lee Turnpenny Reply | Permalink

    Bloody science, eh? Telling people what they don’t want to hear.

    Nice work, Pete. Perhaps you ought (if you haven’t) to have posted a link to this in the comments thread to Hattenstone’s article.

Leave a Reply