Machismo and excellence in cooking and statistics
The inevitable return to TV this week of Masterchef, after a close season shorter even than the English Premier League, has (for strange reasons that I hope nonetheless will become clear) triggered this response of sorts to Brian McGill’s post on Statistical machismo over at Dynamic Ecology last year. Brian lamented the use by ecologists of the latest ‘must use’ statistical method, which is typically complicated both to perform and (perhaps especially) to interpret, without necessarily having much of an effect on the conclusions drawn. He felt this macho posturing – as he puts it, “my paper is better because I used tougher statistics”; in Masterchef terms, “analysis doesn’t get any tougher than this” – ends up overcomplicating papers and wasting everyone’s time.
I enjoyed the post at the time, and felt it raised some interesting points; and though I disagreed with the thrust of it, this was not to the extent that I felt compelled to comment, still less to respond. Now that I’ve come up with a convoluted, almost certainly over-played culinary analogy, though, I’m going to have a bash at expressing my thoughts on the matter properly.
If you watch Masterchef (especially the early rounds) you’ll probably see a great deal of culinary machismo. Even if you don’t, you probably know what I mean: food prepared by someone who is a decent chef, but a pretty awful cook. Smears of jus and droplets of fluid gel on big white plates, but the chicken’s raw; burnt chips in a flowerpot; spun sugar on a duff dessert. Contrast this with what a good non-cheffy cook might produce: a really excellent, well seasoned, ugly stew; a pudding that tastes sublime but looks like a car crash. When I lived in Thornton le Clay near York, our pub specialised in the latter: fantastic, simple, pub food, cooked to perfection with no pretension (it's unfair on them to suggest it was ugly, but the emphasis was on flavour not prettiness). Next village we lived, the pub was very gastro, and the food – though twice the price, and served on wooden boards as likely as not – was nowhere near as good.
This, I think (bear with me!), is similar to the issue that Brian raises. In particular, the use of advanced techniques – statistical or culinary – without having mastered the basics, indeed without even considering the basics, reeks of posturing. In these cases, I agree, we should beware.
Consider for example something like Generalised Linear Mixed Effects Models (GLMMs) as a statistical equivalent of nitro-poached aperitifs or popping candy cheesecake. I am very wary of GLMMs. Ben Boelker’s TREE paper on them basically says as much: do not go here unless you really know what you’re doing. As a minimum, you ought to have mastered the basic component techniques of GLMs and LMMs (and naturally, you need to know your LMs for either of them). Yet I see students who describe themselves as ‘not very confident’ at statistics merrily fitting GLMMs with no clear idea of what model they’ve fitted, or why. Not machismo in this case, but rather blindly following a statistical recipe which demands a great deal more skill than their current aptitude allows.
So yes, in these kinds of cases – and similarly in some of the others Brian mentions – doing a simple analysis well is probably preferable to making a dogs dinner out of a complicated one.
And yet…
Let’s stretch this analogy further. If you really want perfect chips, you’ll triple cook them. Liquid nitrogen really does make excellent ice cream. The way to ensure your meat is exactly à point every time is to cook it sous vide in a water bath. Simply put: some methods of cooking are better than others, and if you can master a Blumenthal-esque skillset, the resulting food will be objectively, qualitatively superior to the lovely, hearty stuff I used to eat in my local, or that I aim to cook at home.
In the same way, some methods of statistical analysis are simply better than others. Brian’s post mentions phylogenetic correction, for example, complaining that it hardly ever affects the result of an analysis, yet entails a great deal of work and additional assumptions. Well perhaps (and his point about errors in phylogenies is a good one), and of course you can fluke the ‘correct’ result with simple statistics, just as you can fluke excellent food with a less scientific approach than that employed by the molecular gastronomists. But if you want consistent excellence – if you want to do something right – you use the best available methods.
Specifically regarding the inclusion of phylogenies in comparative analyses, it’s largely immaterial in my view whether or not this has a large effect on your results; rather it’s simply sensible to consider evolutionary processes when you're modelling a pattern which is the result of evolution. This point is nicely made in a new paper in Methods in Ecology and Evolution by Hernández et al., in which they make a plea for moving beyond phylogenies as ‘statistical fix’ (i.e., ‘phylogenetic correction’) and embracing instead a fully evolutionary view of macroecology in which we test mechanistic hypotheses rather than just describing patterns. (One could of course make a similar case for including spatial processes.)
The cooking/statistics analogy breaks down in one important aspect, however: there are very good reasons why you might not even attempt to master those fancy cooking skills. I read the Fat Duck cookbook much as I might read an account of the building of a great cathedral: full of admiration for the skill, craftmanship and effort involved, but with no intention of even attempting to replicate the endeavour. Blumethal’s Pot roast loin of pork, braised belly, gratin of truffled macaroni, for instance, includes 74 incredients, including two separate stocks (a further 24 ingredients and several hours of prep time), and requires nine separate procedures to produce a single course. You (or at least, I) would never do that to feed two at home; it is only feasible at a restaurant scale. Even those recipes that look technically manageable need expensive equipment, putting them well out the reach of the home cook, who might be better advised to concentrate on mastering more simple skills.
Developing beyond being a good ‘home statistician’ – mastering the essentials of analysis – on the other hand, requires none of this expense. Unlike haute cuisine, mastering statistics – especially in the age of R – is free. We have no excuse not to master the best available methods. So you maybe should roll up your sleeves and chase that Michelin (Fisher? Pearson? Gaussian?) Star after all. Not because you feel you have to in order to show off – I’m with Brian there – but because doing things right is important.
Hi Tom - nice post.
I actually agree with much of what you say. I certainly agree with Hernandez that the goal with phylogenies is to use it to study evolution, not as a nuisance to be corrected. I haven't made the point specifically wrt to phylogenies, but have said the parallel statement repeatedly on spatial autocorrelation.
And I agree that a phylogenetic correction is more correct than ignoring it. However, my point, is whether it is enough more correct to justify its use under some very common scenarios:
a) when you don't have a clue what you're doing - I think we already agree on this - you don't give a creme brulee torch to my brother who has trouble heating a frozen pizza up in the oven, so if he has to eat it will be frozen pizza and that will prevent him from starving (i.e. a simple basic statistic executed adequately lets him move forward).
b) when you don't have a phylogeny - should a reviewer really stop a whole paper when a phylogeny doesn't exist and the phylogenetic correction isn't central to the question and won't change the answer? To use your analogy here, if you're starving and all you have is your good but not chefy pub, should you walk by because its not perfection?
To carry your MasterChef analogy one step further, there may be a clear difference between the output from a MasterChef and your local pub, but its not like there is only one official "MasterChef" way to do things. MasterChef's vary a lot. This is my ultimate gripe in stats. When people take one really fancy method and say everybody HAS to do it that way. By the time its fancy, by the time its MasterChef level, it is almost certainly not the only way to do it.
Brian - many thanks for the comment, I appreciate you taking my post in the right spirit - and apologies if I set yours up as a bit of a straw man, as I think we probably agree more than we disagree!
I think actually your last point is very important, and I'm glad you've made it - in one of Faraway's books (on LMs or GLMs in R, both of which I like) he explains nicely how judgement comes in to analysis such that, even when using the same method (and using it well) different (well-qualified) analysts may apply different criteria and possibly come to different conclusions. So plurality is necessary, and I agree that there are few situations in which there is a single 'right' way to analyse your data. To this extent, a referee insisting on a specific fancy approach is certainly annoying. Sometimes, however, they will be right. Which can be more annoying still…!
On phylogenies in particular, there are workarounds, for example including taxonomic relationships either as analogous to phylogenies or as a factor in analysis, which can capture relationships at a crude level and can help to placate insistent reviewers. But I suspect it won't for much longer as it becomes easier and easier to build bespoke phylogenies on the fly.
I think the age of R point is quite important. It's free to do phylogenetic regressions, and relatively simple once you have a phylogeny. Furthermore it will get easier and more day to day. Any sort of bootstrap statistic was unfeasible 30 years ago, and MCMC is only recently feasible, but now we have the technology. As the analyses become more common they will be put in easier to use wrapper functions.
'it hardly ever affects the result of an analysis'
I also don't like this statement. Non-normal residuals often doesn't affect ANOVAs, etc. You still do it in case it does change the result. And once you've done a phylo regression to check that the results are the same as your LM, you might as well report the phylo regression. I also think the statement is based on the large comparitive studies that have been done; when the analysis takes a lot of effort it's only worth doing if it's the major thrust of an analysis. Now that the methods are easy, you can use them as one, small part of another analysis. This may be smaller and less studied taxa and once the sample size is small the phylo correction is more likely to change the result.
Thanks for the comment Tim. Yes, I think the fact that you can do pretty much any analysis using open source software, and can also freely get a huge amount of data - including that required to build phylogenies - is important. It removes the main excuse for not doing the most appropriate analysis (unlike the £500+ you need for a sous vide machine!) And you're right - if phylogeny turns out to be unimportant, doing the right analysis will tell you that and estimate parameters accordingly; no need then need to go away and do a simpler analysis.
Interesting thoughts. If nothing else, you've made me hungry.
The cooking/statistics analogy breaks down in one important aspect, however: there are very good reasons why you might not even attempt to master those fancy cooking skills.
I think this still holds. You don't need to understand, let alone master, measure theory, but it's the basis of probability theory, and is actively used in several areas of statistical development (e.g. counting processes). You simply avoid that sort of problem (one day I'll spend the time to understand copulas, but not until I think I need them). Similalry, a lot of that Bolker et al. TREE paper is about algorithms to fit GLMMs, but that's something you really shouldn't have to worry about (in teh same way you don't worry about IWLS). You should be able to use the methods confident that they're working correctly: if not, it means my colleagues at the computational end need to do more work.
Good points Bob, although if you'll allow me to stay on the culinary theme, some of those things you mention (measure theory etc.) could be likened to, say, the Maillard reaction - which as a cook it is (arguably) useful to have heard of, but even a top chef really doesn't need to understand the chemistry. So we can take it that there's a lot of computational stuff (or chemistry) underpinning excellence in stats (or cooking); in stats, unlike in cooking, taking advantage of that requires effort but no fancy equipment or expensive ingredients. (I'm hungry now too…!)
I think you are all overlooking one basic fact - from what I can tell from your posts, you all have more statistical training and sophistication than 90%+ of ecologists.
The thought of somebody who doesn't know what they're doing and pulling out R packages to create a phylogeny with no thought into the assumptions or choices the package is making under the hood and then running fancy statistical models under the same conditions doesn't make me feel more comfortable. It makes me feel much less comfortable. Used to be that you had to have read dozens of papers to be considered competent to make a good phylogeny (and I'm just talking about the phylogeny analysis, not the sequencing). Freely available data and an R package that chooses one path doesn't change this.
Please don't give my brother fancy chef instruments!
I agree with much that you write, and with the comments. However, by forgetting opportunity cost you are making the argument a bit to simple. Yes, in the R/Bugs/Python age it is cheap to apply and learn advanced statistics but it still takes time to learn. Time that could be used on new research questions or on learning techniques in other aspects of ecology. Or on gathering new data and gaining field knowledge. Basically, by focusing on learing cutting-edge statistics and refining analyses you are in a sense becoming one type of researcher instead on another one. As a more statistically/theoretically inclined researcher this is easy to forget. You cannot learn everything, and everything comes at a cost (which can be expressed as time). To indulge you analogy - if you only work on perfecting your technique in molecular gastronomy you will end up rubbish at butchering and plating.
The interesting question is where the effort is best spent. This might be on perfecting the analysis and using the "proper" analysis, but as McGill wrote in his initial post, it could also be settling for a simpler analysis that is good enought and moving on to the next problem/project.
That's a really good comment, thanks Tobias. The counter, I suppose, is that with the pressure that we're all under to publish high profile papers, we're all looking to generalise our work. So the field ecologist or experimentalist may have done some great work that can be simply analysed, but is pushed to extend its scope by including some kind of large-scale comparative study too - a study which requires sophisticated methods to be properly analysed. The obvious route then is for increased collaboration between hands-on ecologists and their more quantitative colleagues - this happens sometimes, very profitably, but we should think about how to make it more straightforward, and bidirectional (so that quantitative types have more of an input into data collection strategies too).
Good point, although I'm not sure the solution is simply to accept that people do the wrong analysis - better to focus on training & collaboration. A final return to the kitchen: breadmaking machines are pretty good these days, and allow even the culinarily challenged to produce something - freshly made bread - that until recently was considered rather fancy to be cooking at home. So we need more people like Bob to improve the machinery for others to use to the point where 'PGLS' (for example) can be as confidently pulled off the shelf as 'Anova'.
When writing my earlier post I was also thinking that collaboration is the natural counterargument. And I agree that this is generally a good idea. However, maybe it shouldn't be mandatory if you are clear about your assumptions, the framing of research questions or have made preliminary analysis that indicate that a factor can be safely(?) excluded (e.g. phylogeny)? I guess my main point is that everything comes at a price, and the bar for publication should not be (what some claim to be) perfection, but solid, thoughtful analysis based on clearly stated assuptions. Also, the choice of methods in not only a right/wrong dichotomy (analyses that are plain out wrong is of course unacceptable), but a spectrum balancing simplification, sophistication and complexity. It is also well known that complex analyses can have their own problems, even in the hands professional statisticians, so more is not necessarily better.
I want to make it clear that I'm definitely not arguing for quick and simple though. We should aim high and strive for excellence. I just felt that your original post was a bit one sided (I guess that has never happened in a blog before), and ignored alternative costs and where effort is best spent.
Not sure if I like the analogy to breadmaking machines though. I think "canned"/of the shelf procedures for using complex and powerful methods are generally harmful - and probably more so for advanced methods than ANOVA. But it depends in what level of simplification you are thinking about - drop down menus with click boxes vs. callable functions with good documentation. Some problems with implementation forces users to think about what they are actually doing and this is not a bad thing. I'm sure that you are aware of this though.
You're right of course, there's no right/wrong dichotomy (same as in food…) but I'm pleased that my one-sided post has stimulated such thoughtful comments!
I'm also wary of black boxes, but all of the above was assuming well-documented, scripted analysis (with open code) which I consider essential whether you're calculating a percentage or running something hideously complex - document how you've used the method and at least all is transparent.
Final thought (for now): of course we should be pragmatic in our approach to analysis; but we should strive for excellence at the same time, and encourage it in others too.
Hi Tom. Lovely to see a Masterchef analogy in a science post! It's interesting that you both express some dismay at students/researchers who use high-end statistical methods without understanding them, and yet also herald the recent(ish) arrival of free, easy-to-use software. These things have always seemed to me dangerously intertwined.
As you know I came from a maths background, and during my studies I'd found learning what I might call 'the maths of stats' some of the hardest modules. Yet arriving in your department for my PhD I was amazed at the level of stats that biology students, without any of the same training I'd had, were expected to use (and indeed were using), often without seeming to have a very clear understanding of why or how they were doing it. They just shoved the data in R, pasted in the provided code and out came the results. Now I don't expect that everyone who uses a statistical method should be able to do the whole thing by hand, but there does need to be a basic understanding of, functionally, what the analysis does. I know I had a reputation for R-bashing while I was over there, which is only partially true, but I think R is quite interesting as it seems one of the few command-line (rather than point-and-click) software packages that does easily allow you to perform functions without really knowing how to do it. I think I very much agree with the comment above of Brian here, yourself and many of the commenters are probably at the top end of statistical understanding and so it's fine for you to plough through these analyses, but there maybe needs to be more realism about the training and understanding of many others in the field.
Anyway, I'm happily back in an applied maths cocoon now...
Thanks for the comment Alex! You are of course correct that R allows the incautious user to do any number of things that they really shouldn't, and that does conflict with the need to understand a method before using it. On the plus side, it does (if used properly) leave an audit trail of what decisions you've taken (something which really ought to be required to accompany every paper in my view, regardless of the sophistication and complexity of analysis).
More generally, while I agree that a good knowledge of the theory underpinning any analysis is very useful, most of us take a great deal on trust - we trust the judgement and advice of our colleagues, supervisors, whatever, whether that be at the level of measure theory (cf Bob's comment above) or of implementing the method du jour. This is not restricted to analysis of course, and is probably a necessary element of progress, because one person simply doesn't have the time to master everything from first principles. I can see how this can lead to the perpetuation of bad practice however, and am not sure how best to avoid it.
And regarding my own level of statistical sophistication, that's generous of you - personally I think I'm a better cook than analyst…