The pseudoscience of anecdotes
The results from the TREAD study (TREAtment of Depression with physical activity) came out today in the British Medical Journal – you can check it out here. TREAD was a long term randomised controlled trial (considered the ‘gold standard’ in clinical testing) which aimed to figure out whether giving people who were suffering from depression a bit of help with their exercise regime (in other words, having a facilitator provide advice and encouragement to the patient) actually improved their mood or not over time.
It would be great if it did – it would mean that we’d have an alternative to drug-based therapies that would be simple for doctors to prescribe, and desirable for patients who were worried about the side effects of antidepressants. The trouble with the results from TREAD, though, is that they didn’t show any benefits for those who received the intervention, when compared to those who didn’t – either in terms of improved mood or reduced antidepressant use. In fact, the only thing that did improve was the intervention group’s amount of physical activity during the follow-up period.
If his article in the Guardian today is anything to go by, this seems to be a huge irritation for Simon Hattenstone. The article, in fact, provides a masterclass in what you should look out for in a poor piece of science journalism. Let’s have a look what the problems are.
“So all of you who thought you were learning how to cope with your out-of-kilter brain, who had worked so hard to release endorphins and get a serotonin surge, who had made life manageable by running, going to the gym, dancing, or whatever, were WRONG.”
Making out that the the study is telling people who are depressed that if they’ve been exercising, it’s a waste of time, is a good way to get readers on side. It tends to rile people up, and make them amenable to the point that the writer wants to make.
“According to this report, carried out by the universities of Bristol and Exeter, and funded by the Department of Health, you exercise-tastic depression-battlers are simply deluding yourself. The study is, apparently, the first large-scale, randomised controlled trial to establish whether exercise should be used in primary health care to help treat adults with depression.”
Or, y’know, just spell it out in no uncertain terms – ‘this report is telling you that you’re stupid if you’re trying to make yourself feel better with exercise.’ It’s got good shock value. Also, note the use of the word ‘apparently’ in the second sentence – it makes it sound like the research is trying to make itself sound more important than it actually is. For reference, the actual words from the BMJ article are ‘the trial is one of the largest trials of physical activity and depression to have taken place in primary care.’
“As a depressed manic exerciser, who has found running hugely helpful, I would like to blow a great big Panglossian fart in the face of this churlish research.
Writers who have a drum to beat tend to identify themselves as authorities-from-experience on some subject or another. It lends weight to any points they make because they can say “I’ve been through that, and I know that X actually does work for me”. And actually, that’s a perfectly valid opinion to have – I don’t wish to belittle anyone in this way at all. The trouble is, it’s not particularly scientific. For example, I’ve suffered from moderate-to-severe depression over the past fourteen years, and I’ve often taken to the gym in the hopes that it would help my mood. It’s never worked. Actually, at times, it’s made me feel worse about myself, because I could never stick to it. Does that mean that exercise doesn’t work for depressed individuals? Not at all, it just meant that it didn’t work for me. It’s in this way that arguments from experience are extremely compelling; the ones it works the best on are those that make the arguments in the first place. As we’ll see later on, it’s also why they’re potentially the most dangerous sort of evidence.
First of all, slightly less depressed, even if not statistically valid, has to be better than the same or more depressed."
And here we get to the crux of the problem. The results from TREAD found that those in the intervention group were very slightly less depressed at the end of it all, when compared to the control group, but this difference wasn’t statistically significant (or, in the article, ‘valid’). ‘Significant’ (and ‘valid’, but for different reasons) is one of those words that means different things for scientists and the public. In everyday terms, if something is significant, it means that it’s really important – for example, the European Championships is a significant tournament in the footballing calendar; it’s a huge honour if you even get picked to play in it, let alone win it. So when you say that something’s not significant, like a result from a study in which some people become slightly less depressed, the impression you get is that of a cold and heartless scientist; someone who doesn’t care that a patient got a bit better, because it ’didn’t fit with the rest of the results’.
Significance in the scientific or statistical sense, though, doesn’t mean this. If something is statistically significant, it means that it is unlikely to have occurred because of random, chance factors. What that means for the TREAD study is that, yes, some people did get a bit better, but we can’t be sure whether that’s because of the intervention they were given, or because of something else that happened to them. In this way, it’s a form of caution. It’s great that some people got better – it would be great if even more did – but if they didn’t get better because of the treatment, then there’s no point in rolling that treatment out (at great expense) across the NHS.
"Second, it is unclear, to me at least, what stage of depression the participants were at. Yes, if somebody was so depressed that they can’t face getting out of bed, ordering them to go for a 10km run probably wouldn’t do the trick. Indeed, it might make them feel considerably worse.
This is actually a fair point – it’s really good practice to critique studies in this sort of way. We have to be sure that the people being tested are appropriate for the study. Moreover, it’s difficult – as the article states – to compare how the intervention might affect someone who is diagnosed with dysthymia to someone with severe depression. You need to try and keep your patient population as similar as possible to each other in lots of different ways, so you can be more sure that the results you get are because of the intervention you’re providing, rather than because of individual differences in the patients. From reading the BMJ article, it looks like the researchers performed a pretty comprehensive baseline screening session, and had quite detailed exclusion criteria. That doesn’t mean that it’s perfect; scientific studies never are. But the right sorts of protocols seem to have been followed.
However, we depressives don’t “use” exercise like this – it’s not a panacea, it’s a means of managing depression once we’re beginning to feel better or when it is in remission; a way of keeping it at bay."
So just when you think the article’s getting better, your hopes get completely undermined by this sort of statement; lumping ‘depressives’ together into a characterless group, defined only by the fact that they’re all depressed. Making a point about how it’s important to consider individual differences, and then completely ignoring your own advice straight away is an automatic fail and renders the rest of the argument redundant.
I’ll finish by going back to the points about arguments from experience, summed up nicely in the final sentence of the article:
Perhaps we should rely on self-knowledge rather than research when it comes to depression. After all, nobody knows your own body and mind quite like you do. So sod the academics, I’m off for a run.
No. No no no no no. No. The absolute last thing that we should be doing is relying on self-knowledge. If you rely on anecdotal evidence, then health policies (any sorts of policies, actually) just become a shouting match where whoever yells the loudest gets their way. Sometimes, when I get depressed, I eat loads of chocolate. Sometimes it makes me feel better. If we were to appeal to self-knowledge, then on the basis of my ‘evidence’, the NHS would be sponsored by Cadbury’s and we’d be paying £7.40 for a bar of Dairy Milk at the pharmacy. And weirdly, it might actually work for some people. The thing is, other treatments would work better and for more people. So for the vast majority of people, eating chocolate would not only be a monumental waste of time and money for everyone involved, it would also have the more threatening implication that people would not be having the proper treatment that they should be getting – the one that actually works. The only way that we can figure out what those treatments are is through randomised controlled trials and solid, objective research.
So, what we need to take home from the TREAD study is that, while having someone support you and provide advice on your exercise regime might help some people with depression, it doesn’t seem to work for most – so at this point in time, it’s probably a good idea not to spend loads of money rolling it out across the country. We need to look at it again, and think about why it’s not working (and who knows? That sort of research might find something that does work for everyone). What we shouldn’t be doing is writing sensationalist articles that misrepresent both the study in particular, and science in general. It doesn’t do anyone any favours.