Wild extrapolation and the value of marine protected areas

23 July 2013 by Tom Webb, posted in Uncategorized

Last week, the UK National Ecosystem Assessment published a follow-on report on the value of proposed marine protected areas (rMPAs) to sea anglers and divers in the UK. This report gained a fair bit of coverage, likely because the headline numbers it proclaimed are quite astonishing: “The baseline, one-off non-use value of protecting the sites to divers and anglers alone would be worth £730-1,310 million… this is the minimum amount that designation of 127 sites is worth to divers and anglers”. Furthermore, they claim an annual recreational value for England alone of the rMPAs of £1.87-3.39 billion, just for these two user groups (divers and anglers).

These numbers are so astonishing, in fact, that my bullshit klaxon went off loud enough to knock me off my chair. See, I’ve been thinking recently about sea angling as an ecosystem service, and so know that there’s estimated to be somewhere around 1-2 million sea anglers in the UK. The number of divers is, I reckoned, likely to be considerably lower (there’s a higher barrier to entry in terms of equipment, qualifications, etc.). So these headline figures imply an annual spend -  purely on their hobby - somewhere in the order of £1000 for every single self-declared sea angler or diver. Which seems rather on the high side, given that one would expect a very long tail of ‘occasional’ dabblers in each activity (e.g. people who spin for a few mackerel on holiday).

So, I bucked down and read the 125 page report, to find that the authors had done some things really nicely. Their valuations are based on online questionnaires featuring a combination of neat choice experiments, willingness to pay (WTP) exercises, and an valuable attempt to characterise the non-monetary value of the sea-angling or diving experience (things like ‘sense of place’, ‘spiritual wellbeing’, etc.). But the headline numbers are highly dubious (worthless, in fact), because they did a few things very very badly indeed. Unfortunately, they did a different bad thing for each of their two major monetary valuation methods, so the numbers emerging from each are equally dodgy, as a modicum of mental arithmetic, common sense, and ground-truthing will show.

First, the annual recreational value models are nicely done, using a choice experiment based on travel distances to hypothetical sites with different features to assess which of those features are most valuable. Mapping these features onto the rMPAs leads to a ranking of these sites in terms of how attractive they are to anglers and divers. One could quibble with details here - perhaps the major quibble would be that there is no ‘control’, i.e. no assessment of the value of sites which are not proposed for protection. But in general, I think this analysis gives a decent estimate of how the survey respondents value the different sites.

They then attempt to get an overall annual value for each site by multiplying its value to individuals by the number of visits it receives in a year. This is where the problem arises: attempting to generalise from these respondents to the entire population of anglers (estimated at 1.1-2 million) or divers (estimated at 200,000). I’m going to concentrate on the anglers because the issue is most extreme here: their models are based on 273 responses, a self-selected group of anglers acknowledged within the report to be especially committed (averaging 3-4 excursions a month) and interested in marine conservation, and representing between 0.01 and 0.02% of the total population, i.e. 1 or 2 responses per 10,000 anglers (they also a self-selected sample of highly experienced divers, representing around 0.5% of all divers, i.e. 5 per 1000). Extrapolating from this sample to the entire UK angling population produces some interesting results.

For example, using this methodology Chesil Beach & Stennis Ledges rMPA on the Dorset Coast has an estimated 1.4-2.7 million visits by sea anglers annually. That translates to 3800-7400 visits every single day of the year. Compare this to a (highly seasonally variable) average of around 3000 visits per month to Chesil Beach Visitors Centre. Or you could look at the Celtic Deep rMPA, a site located some 70km offshore, where they estimate between 145,000 and 263,000 angling visits per year. That’s 400-720 visits a day, which translates to approx 40-70 typical sea angling boats, each full to the gunwales every single day of the year. Of course, this is simply because the tiny sample is uncritically extrapolated. In the case of the Celtic Deep, it is straightforward to calculate that there were actually 36 observed visits, which (when divided by 273 and multiplied by 1.1 or 2 million) gives you 145-263,000 estimated visits. Using this logic, the minimum number of visits a site could receive is (1/273)*1.1 million, or >4000. Diving numbers are similarly unrealistic, with estimates of 123-205,000 visits a year (340-560 per day) by divers to Whitsand & Looe Bay, or 26-44,000 a year (70-120 per day) at Offshore Brighton, which is around 40km offshore.

This kind of wild, uncritical extrapolation is staggering, akin to using the opinions of a focus group of LibDem party activists to predict a landslide in the next election. It’s a textbook example of the utility of a bit of simple guesstimation (e.g. a million visits a year means 10,000 visits/d for 100 days, or ballpark 2500/d over the whole year), allied to some common sense (have you ever experienced those kinds of numbers when you’ve visited the UK coast?)

So, we can discount the big annual recreational value figures. What about the WTP exercise? WTP has its fans and its critics. My view is that it’s a useful way of ranking scenarios according to preference, but I don’t give a lot of credence to the ££ generated, simply because by increasing the number of scenarios you can quickly get people to commit more cash than they intended. But regardless of that, the authors of this report appear to have made a very strange decision in aggregating the WTP estimates arising from their questionnaire. They worded the questions very carefully, presenting each respondent with a single site, outlining its features, and asking how much they would be WTP as a one-off fee for its protection - being sure to think of this amount as a real sum of money, in the context of their household budget. These numbers are then used to give an average WTP for all the rMPAs, which seems reasonable, and a useful way to rank the sites.

But They then simply multiply these site level averages by the whole UK angling (or diving) population to get a total WTP for the whole set of rMPAs.

Think about what they’ve done there.

They’ve asked people how much they would be willing to pay to protect a single site, and have then assumed that the same person will pay a similar amount for every site in the network. So if you agreed that you’d be prepared to pay a one-off sum of £10 to protect a site, you could find yourself with a bill for over £1000 to protect the whole network. (This is a slight over simplification, as specific values are site-specific, but it is essentially what they’ve done.) You simply cannot aggregate WTP like this. I mean, I’m not an economist, but if economists think you can do this, they are deluded.

Again, a bit of common sense would have helped here. The authors compare this WTP to an insurance premium, which is a useful analogy. But how many anglers or divers are really, when it comes down to it, prepared (or even able) to shell out a £1000+ insurance premium to prevent damage to the marine environment which may or may not occur in the future?

Anyway, that’s what’s been bugging me these last few days. I could go on (for instance, on a more philosophical level, is replacing strictly regulated commercial fishing with unregulated recreational angling necessarily a good thing for the marine environment? Will diving or - especially - angling actually be allowed in these rMPAs?). And there are some useful things in the report. It confirms that people do value the marine environment, really quite highly, and that different features are valued differently by different groups - a useful starting point for some more focused research, and helpful in placing relative values on different rMPAs. But unfortunately - inevitably - media attention has focused on the ludicrous headline numbers, something the authors have actively encouraged in their framing of the report.

A final positive point to end on: my bullshit klaxon seems to be in fine working order.

11 Responses to “Wild extrapolation and the value of marine protected areas”

  1. Jasper Kenter Reply | Permalink

    Hi Tom. This is the lead author of the report here.
    As to the recreational values, yes- these are guesstimates, particularly for anglers, because of the uncertainty of extrapolating up to the entire population. For divers we had about 1200 and extrapolating to the 200.000 or so divers in the UK is probably quite reasonable, but extrapolating the 400 anglers to the 1-2 mln is indeed more tricky, particularly for places like the North of Scotland where concentrations are very low. However, the report fully acknowledges these limitations, in detail, everywhere where total figures are presented. In fact, the report says that these recreational figures should be read as indicative, not as certain figures. I am not sure, if you read the report, why you didn't mention that in your blog.
    Re. the non-use values based on donations. We purposely used a one-off donation for this reason. If over the next 20 years, people could be asked to donate to many places one a one off measure. The average donation was around a fiver, though in our way of calculating, this is far lower if the site would be far away from you and potentially higher if it is near. We also account for those with a lower income donating less and other factors like that. Costs and benefits in the impact assessment of MCZs by Defra is taken over a 20 year time period. So each hypothetical donation to each site is counted only once over that period of time. So yes, on average £1000 per head, but over 20 years. I.e. MPAs would be worth £50 per year to divers and sea anglers, on average. Hope this calms down your bullshit claxon :-)

  2. Jasper Kenter Reply | Permalink

    Woops, some sentences went weird there. Ignore the if in 'If over the next 20 years, people could be asked to donate to many places one a one off measure'

    By the way, in my view indicative figures are better than having no figures if no figures means that benefits are not accounted for. Valuing things that fall outside of the market is always a tricky task!

    Also, our press release http://www.abdn.ac.uk/news/details-14033.php, and many media publications, did pick up on that there was uncertainty over figures because of uncertainty over visitor numbers.

  3. Tom Webb Reply | Permalink

    Hi Jasper,

    First, many thanks for commenting, I really appreciate you doing so in good humour - and sorry it's taken me all day to sit down to respond! I did of course read the full report, and really liked your approach in much of it - I should have made that clearer in the post. In fact, I thought your methods were really quite innovative, right up to the point where you extrapolate to get aggregate values. But I'm afraid I still can't see how these are defensible in any sense.

    On your first point, I appreciate that you emphasise sources of uncertainty both in the report and in the press release. However, it is rather disingenuous to suggest that this makes it all OK. In your report, the 'headline results' give the monetary values up front, on p4; the first mention that we should actually consider these results (presented in £) to be 'indicative only' occurs low down on p5, and not until p6 do you mention that we should actually consider them to be 'relative trends'. This pattern is repeated throughout the report: present the numbers first, as if they mean something in absolute terms, and only later add a cautionary note. And saying there is 'uncertainty around' the angling figures is also misleading: I am certain that they are not underestimates! More generally, if numbers are suitable only for relative ranking of sites, they should be unit-less. As soon as you add the £ in front, people will quite reasonable assume that they refer to actual monetary values.

    Re. your sample sizes, the angling sample is clearly inadequate for extrapolation, and I'm less confident than you that the diving sample is representative. Sure, it's a larger proportion of the total population, but it's also self-selected, and I doubt these committed divers are representative of the 200K BSAC members in terms of experience (half had >200 lifetime dives) and numbers of dive trips per year (47 days/y is a big commitment). In all these kinds of sports clubs, there's a very skewed distribution of activity by members: only a handful turn up week in, week out. I did exactly one trip as a BSAC member, for example! In my experience, these passionate few will be exactly the kind of people who will volunteer to fill in quite a detailed questionnaire about their hobby. Or to put it another way: do you honestly believe there were 300+ divers at Witsand & Looe Bay today?!

    Regarding the aggregation of non-use values, I take your point about it being a one-off donation for 20y, but your question does not ask this. Instead your ask "If you were asked to make a one-off donation to support protection of site x into the future, how much would you be willing to donate? Please carefully consider the characteristics of site x. Your donation would be used to set up a local management trust to maintain this site as it is shown below, and protect its natural features against the risk of future harm and degradation." Taking the response to this, and assuming the same person would be prepared to multiply their donation to protect multiple sites, does not seem reasonable to me. Or put it another way: if you'd asked people to pay a one off donation to protect all sites, do you think they would have committed >100x what they did for one site? Seems doubtful to me.

    Are these indicative figures better than having no figures at all? Maybe, in the short term. Longer term, you risk losing credibility if the indicative figures turn out to be orders of magnitude off the truth.

    Anyway, aside from the wild extrapolations, there's an awful lot of very stimulating things in your report, very relevant to stuff I'm thinking about at the moment, so thanks!

  4. Jasper Kenter Reply | Permalink

    Ok, just a quick response because I have the next NEA report to finish by tomorrow!

    - Thanks for the compliments on the research design for the choice experiment and contingent valuation

    - Re. the uncertainty. This is made abundantly clear, far clearer than in most reports. You are also not accurate. In the first paragraph of the headline resutls the main results are listed. Already in the second paragraph it is mentioned that "figures come with a number of limitations (see below)". Immediately after in the next few paragraphs the remaining results are presented is the uncertainty specified in more detail. This is actually very uncommon for headline results, often these don't mention uncertainties at all let alone in any detail. Our report has been very honest and it is disingenious to suggest otherwise.

    - You are also not being accurate about what the report says about this. It is written that the recreational values of anglers should be read as uncertain and indicative because of the sampling issues and uncertainty re. visit counts. The *individual* site results should be read as relative trends, giving some indication as to which sites will be worth more in terms of recreation, but where there are obviously some sites where the figures won't hold. However, at a national scale, since errors can be in both directions there will be some cancelling out, and I believe the order of magnitude will hold, which is why we used the word indicative.
    For the non-use values, and the divers recreational values, these are likely to be underestimates, and a long list of reasons for this is given in the discussion section of the report.

    - Re. non-use values. Again, you do not accurately reflect how the aggregation worked. It wasn't simply a matter of adding all the sites up. We used a GIS to consider distances between each sample member and each location. On this basis, everyone does have a value for each site, but this gets lower and lower the further away the site is. This is analogous to, if you were asked you might be willing to give a donation towards helping set up an MPA in the Shetland islands, but you might only give a pound if you live in Southampton and at the same time give £20 for the Isle of Wight. We only ask for one site at a time (we actually asked each respondent about four sites in sequence), asking about 155 sites in one go would be absurd and it wouldn't allow us to make site-by-site estimates. The procedure we used is perfectly accepted in economics and the methodology and reprt has been peer reviewed by one NEA expert panel reviewer, two external reviewers, the NEA co chairs and several government reviewers (names are in the acknowledgements). You can ask for the reviewers comments and how they were responded to from the NEA secretariat, if you can be bothered, without an FOI request...

    - Non-market valuation is a tricky area and it is inherent to this that estimates will have considerable uncertainty around them. However, without it, at least in economic analyses we can not account for the massive benefits that the environment has to people. As it stood before this report, in the impact assessment for the English MCZ consultation, there was a massive gaping hole in the page for the economic benefits of MCZs while there was a long list of costs. Very difficult to defend a policy if you don't have evidence on benefits and you are pushed in time of austerity to bring down any costs you can. Hence only 31 of 127 English sites are nominated for designation and even those are yet to be confirmed. Thus admittedly uncertain evidence on the benefits of MPAs is far better than not having any and simply not accounting for benefits.

  5. Tom Webb Reply | Permalink

    Jasper - thanks again for the thoughtful response. My worry is that anyone looking in detail at the visitor numbers you produce will come to the same conclusions that I have. Do you really believe they are reasonable estimates? Personally I would need much more substantive evidence before I accept that there are more diving visits made in England each year than there are on the Great Barrier Reef (which is what you're claiming). And everything in my own experience of our coasts screams at me that your angling numbers are huge over-estimates - do you have numbers e.g. from major angling festivals you could use to ground-truth some of these? Extrapolating from a small, non-random sample to a whole population is basically never justified, and I'm surprised the many reviewers didn't pick up on this.

    On the WTP results, OK - as I said, I'm not an economist, and if this is how things are done so be it. But as a potential respondent to these kinds of questionnaires, asked to consider real sums of money, were I to agree to an amount to protect a very small number of sites without being told that I would then be expected to stump up for a much larger number (albeit distance decay corrected etc. etc.) I would be disgruntled. Again, great for determining relative values, but leave off the ££ in the results!

    Also, regarding benefits of MCZs, a point I glossed over in my post is that you have not compared the proposed sites to any other sites, which is important in the interpretation of your recreational value results - are the MCZs more or less valuable to divers and anglers than any other (potentially cheaper to protect) sites? Surely this would need to go into any formal cost-benefit analysis of rMCZs? But good on you for attempting to fill the gaps on benefits, and good luck with the next report!

  6. Tom Webb Reply | Permalink

    I should have emphasised in the above comment, btw, that the >2 million predicted dive visits (and order of magnitude more angling visits) are *only* to proposed MCZs in England, not the whole of the English coastal environment. The 127 rMCZs form a large network, but cover a fraction of our coastal environment (I think around 37,000km2 out of several hundred km2, hard to find exact figures for England only) and exclude some key dive sites like the Farne Islands (see http://www.mczmapping.org). So predicted numbers of visits for England as a whole would have to be on the order of at least 10x the numbers presented in the report, unless preferences for sites in the network are really very strong among divers and anglers (which they may be, but I'm not aware of any evidence to suggest they are).

  7. Jasper Kenter Reply | Permalink

    Tom - very quick reply as my nxrt report is still far from finished. The report estimated off the top of my head 19 visits by divers to the >150 sites as a whole, per year, based on the stated visits by the sample (with more than 1 of every 200 divers sampled). At some more popular sites people go out with two dozen on a rib with sometimes 25 boats out. We've heard this for several sites for some of the workshops we;ve been running. So yes, it is really possible to have hundreds of peolpe at a site in a day. Many sites are also large, so if you go there as an individual and dont see many others, it may be quiet in your subjective experience, but there may still be many others out.
    Predicted nrs across the country would not be that much, because a) many of the proposed MPAs are key dive sites and b) even though potential sites cover (again off the top of my head, I have no time right now to look it up) 12% of the UK sea floor much of that sea floor is in deep water whilst most of the area of the sites (and all the sites we looked at) is/are in shallower waters, so the actual proportion is probably more like between 1/4 and 1/3, but I havent calculated that exactly.

    Comparing with other sites not proposed as MPAs - doesnt matter, because we would just look at their features, which is what predicts their value. Because they wont have many of the conservation features that proposed MPA sites have (theyre not proposed for nothing...) their value will probably be substantially lower, but (because of the way we use the GIS in estimating values) it will also depend on how many anglers or divers live in the vicinity, the more there are nearby, the higher the value and vice versa.

    I am glad this is kicking off a bit of debate, because what really needs to happen is more research into what divers and anglers are actually doing. This piece of research is more of a side-product of the NEA, our project http://www.bcu.ac.uk/research/-centres-of-excellence/centre-for-environment-and-society/projects/shared-values is looking at shared, plural and cultural values with divers and anglers as a case study, and we needed a baseline of individual values, before we were going to run workshops to look at how people express values as a group. So we didnt exactly have the time, resources or scope to look in depth at divers and anglers activity at sites around the UK.
    There are almost no reports for sea anglers and divers, so it was actually extremely difficult to compare our visit counts to any other sources. Sources that the govt uses at the moment are all anecdotal, i.e. some back-of-the-envelopes by stakeholders, local MMO officers etc.
    So it would be great to get to the bottom of this. The nice thing about the methodology we used, is that if there are updated, more accurate visit counts, we can update the figures.
    In the meanwhile, I stick to my point that having some figure is a lot better than none. Even if visit counts would be overestimates, there are many other factors that push the other way. E.g. we weren't able to include charter and other boat costs in the transport cost calculations, which are of course far higher than car miles.
    By the way, the recreational values (which are based on visit counts) are not actually used for impact assessment of MPA/MCZ policy, because they are current values and it is hard to say how they will change as a result of introducing new MPAs. Only the non-use values would feed in. And they aren't dependent on visit counts. And that's also the figure we pushed more in the media talks.

    If you're looking for perfect figures, don't look to econonomic ex ante estimates. Even if based on market prices rather than non-market valuation they're rarely accurate. Personally I think we rely on economic analyses too much in decision-making, but that's a whole different discussion (am I digging my own grave here...)

  8. Tom Webb Reply | Permalink

    Jasper - I also have a ton of things I should be doing, so this'll be quick too! Re. visitor numbers - of course, you can get large numbers of divers at a site. This weekend for example, if the weather holds, 25 boats x 25 people at a site seems reasonable. But in stormy November? And on subjective experience, I chose the Chesil Beach example as you can stand on high ground and see the whole expanse of beach. 1000s of anglers would be obvious. And if you're predicting more people daily than typically participate in, say, Scarborough Angling Festival, I'd be suspicious. There's a lot to be said for the back of the envelope!

    Of course, I'm not looking for perfect figures. Valuation is important (for better or worse!) and all valuations will be wrong to some extent, just like everything I estimate as an ecologist is wrong. You mention some underestimates in your figures; I don't think any of these will be orders of magnitude out, which is what I think your visitor numbers are, because you've extrapolated from a biased sample. As I've explained above, I worry about your non-use values too, because it seems that 'or's have been switched to 'and's. I care a lot about marine conservation and hope that your headline numbers don't get picked over elsewhere and distract from the key message that anglers and divers really do value the marine environment.

    Great to know you're working more on this though. I have a coastal ecosystem services project running, mainly working with commercial fishermen at the moment, but we're looking to work with sea anglers too - perhaps we could correspond more positively at some later date? :-)

  9. Jasper Kenter Reply | Permalink

    Just a final reply on the or's and and's in relation to the non-use values. I already mentioned that the way we did this, using GIS based aggregation, is increasingly common and generally accepted in environmental economics as a sound approach. In most cases valuation studies look at the marginal benefits of a single site or single project. In the past, in contingent valuation studies people from a fairly arbitrary political area (e.g. the county that the project would take place) would be asked for their WTP, and then the values for that would be aggregated. Nowadays, however, we assume that people in far wider units (e.g. all of the UK) might have values for that place, though they are smaller than those of locals because people from far away dont use the site, or at least not very often, so their main value would be the value associated with just knowing that that place, or those species would be protected ('existence value'). So now, increasingly a GIS approach is used where values decrease with distance.
    We could in theory do 150+ studies across the UK to ask everyone about each possible MPA individually, asking them how much people would be WTP on an annual basis for that site. That would be the ideal, but it is practically impossible. But even if we did that, we would still have to add up the sums if we wanted to know the total value of introducing the 150+ sites as a network. (We should then really also look at the added value of the network as a whole - something we werent able to in our study, another limitation that suggests our figures are an underestimate, but that aside).
    Of course, doing 150+ studies is practically unfeasible, so we have to use a value transfer approach, as we did, by asking people about hypothetical sites with certain key characteristics (e.g. habitats, species, access, restrictions) and then transferring the outcomes to actual sites, on the basis of those characteristics, and on the basis of how far away the site is to populations of divers and anglers, which is what we did, as you know.
    If we used the first project-by-project approach with different research projects, we would still ask people about one site and then add it all up together.
    It is fair to assume that people would be willing to pay for a whole series of donations over time, and in reality those donations would be considered on their own merit, rather than a series, though there will be some interdependencies, e.g. anchoring ("I paid £10 last time so I'll pay about the same now") and allocating payments to a budget within someone's household budget ("I spend £20 a month on charity and I paid £10 to the red cross, so I won't pay more than £10). If we would ask everyone to pay for 150 sites from their current or current annual budget, that wouldn't accord with how people actually operate. It makes more sense to consider what people would be willing to pay for a number of sites (we asked each participant about 4 hypothetical sites), and then use that to see how distance, site characteristics etc. influences how much people are WTP, and then extrapolate that over time but under the assumption of a long, rather than annual, timescale.
    Our overall non-use values if you add it all up weren't actually that big if you compare it with other studies that looked at non-use values but for a possible MPA network as a whole, a couple of years ago (see final discussion in the report for an overview of other studies).

    Re. further research. This was part of an externally funded project, and as I mentioned almost a side project of the NEA, and we don't have any further research planned into recreational activities/frequencies etc. at the moment. However, I strongly hope that Defra, and the devolved governments, will put some more resources towards getting a better idea of recreational activity around the coast, and how marine protection might affect this. Hopefully this report and the 1000s of words written in it that were dedicated to explaining limitations and data uncertainties might give a bit of a push!
    Thanks anyway for the lively debate.

  10. Tom Webb Reply | Permalink

    Jasper - thanks again for commenting, the debate has been interesting and I think fruitful! My piece in The Conversation (http://theconversation.com/cash-from-conservation-zones-doesnt-add-up-16378), although only just published, was written last week, so doesn't address your responses very well - apologies for that, although I stand by my main points.

    On WTP, then, I appreciate the sophistication of your approach, and I think you have a nice indication of the rank order of site values, but I remain sceptical that WTP is additive in any simple sense and therefore that the aggregates mean very much in terms of actual ££. My point wasn't that you should have asked people about 150+ sites separately - clearly that would have been impractical for you and would have lead to massive cognitive overload for participants. However, it would have been nice, had your sample been big enough, to split respondents, and ask some the question as you did, but ask others to value a set of sites en masse - e.g. 'how much would you pay to see all the proposed East of England sites protected / managed' or even 'how much would you pay for the entire network to be protected' - this seems reasonable, given that the MCZs will be designated centrally. My suspicion remains that values for sets of sites would not scale simply from values for individual sites. This is something Defra etc. could usefully put some cash into finding out!

    The second problem also remains - i.e. you're extrapolating from small, self-selected samples. It's simply very hard to say anything very useful about whole populations based on that.

  11. http://tinyurl.com/ Reply | Permalink

    I every time used to study post in news papers but now as I am a user of net therefore from now I
    am using net for articles or reviews, thanks to web.

Leave a Reply

× nine = 72