Confessions of an Open Access Agnostic
The office that I worked in a few years ago had a window that opened onto the main University of Sheffield concourse. Every so often, lunchtimes would be enlivened by a student protest (typically over fees), during which someone with a megaphone would shout a lot. I remember clearly being struck with the thought, “I wonder if anyone has ever changed their mind about anything as a result of something they heard through a megaphone?”
It certainly doesn’t work on me. Even if I broadly agree with the shouty person, the louder they shout the more inclined I am to pick holes in their argument. It is a character trait of mine, I’m not sure if you would call it a flaw or a virtue, that I hate being told what to do and, especially, what to think.
All of this explains, perhaps, my ambivalence towards Open Access (OA) publishing. I don’t like being told where I can and can’t publish. I distrust zealots, including well-resourced single-issue campaign groups which will hear no alternative views, which present shades of grey as simple black/white dichotomies, and which (a pet hate) bandy around variants on that tabloid favourite ‘tax-payers’ money’ (when they mean ‘public money’). I worry about people being pressurised into publishing in inappropriate journals, or – if they decide to stick with a non-AO journal, for whatever reason – not receiving the quality of review they deserve because of misguided boycotts. I don’t appreciate non-scientists in the media wading in with their ’aren’t you silly, you’ve been doing this all wrong for decades’ line. And I’m wary of the creeping sense – by no means restricted to science – that content should always be free, regardless of the costs involved in producing it. I’m not comfortable with the big publishers making huge profits from the outputs of science, but I also recognise that good publishers (and their employees) have done, and continue to do a terrific job to ensure the effective communication of science.
Of course, there is a more nuanced debate going on underneath the bluster. From what I see on Twitter, today’s debate at Imperial seems to be a good example (#OAdebate). Some very clever and thoughtful people have weighed things up and come down on the side of OA. And I’m not even sure that I don’t agree with them. Certainly, I am all in favour of the broader Open Science agenda – opening up the data we produce, and the tools we use to access and analyse it. But I remain to be convinced that access to primary research papers is such a big issue that it should be pushed above all else (partly because, with a bit of effort or an email or two, it’s usually possible to access most recent papers), and that all of this energy should be focused on it (whilst overlooking the interesting and potentially profound financial and sociological implications for scientists and their institutions).
My beef is not at all with OA, but rather in the way that the debate has been framed in terms of good and evil, right and wrong (not a million miles from the ongoing GM debate). Subscription-based (reader pays) publication of publicly-funded research costs public money, and has pros and cons. OA (author pays) publication of publicly funded-research costs public money, and has pros and cons. A shift to OA will not (I’m pretty certain) be accompanied by an injection of new cash, but will rather see a shift from funding infrastructure (especially libraries) to funding individuals (e.g. through research grants). And the debate should be on how best we spend limited public money to communicate the outputs of research in the most effective way. It could be that making all primary research available to everyone is the way to do this (although I don’t think accessing papers is quite so difficult as some would have us believe; and in any case, the readership for the vast majority of papers is tiny). It could be that we’d be better advised to concentrate on more effective communication of key results in other formats, or in making other products of our research (especially data) more widely available. Even if we hold OA as something to aspire to, I feel that blindly pushing it as a top priority risks sidelining more important debates about opening up science.
So thanks, but I won’t be signing any petitions just now.
I realise the focus of this blogpost is OA, but I take issue with your assertion that the OA and GM debates are similar in being cast in terms of good vs evil. I agree that some people promote OA as good and for-profit publishing as evil, and I’m glad you point out the grey areas. But the current Rothamsted wheat trial GM debate involves many who accept it is not a black and white issue, it’s more some seeing grey and some seeing red. There are many people of various backgrounds eager to have an open, rational debate about GM crops, which is admittedly a contentious topic where some are for, some are against, and others are undecided. This diverse group has begun a petition in response to threats to destroy an entire field experiment. I might not sign a petition for OA, or boycott a publisher, but I would seriously consider signing a petition to stand against violent protest and acts of vandalism that would annihilate some scientists hard work and stand up for the debate and testing of ideas.
Charlie – thanks for pulling me up on what was a rather facetious comment, and in no way do I want to equate pro-OA with anti-GM. However, I maintain that there is a parallel, in that both debates involve entrenched ideologies, and both involve a committed campaign group (convinced of their righteousness) against a large and profitable business with a big PR problem.
Also, put yourself in the position of a GM protestor for a moment and the Rothamsted position doesn’t seem so reasonable – what were the chances of them (Rothamsted) deciding to scrap the trials following the ‘open, rational debate’? If there was no chance of them changing their mind, why would you (as an anti-GM activist) be interested in participating? I heard a senior Rothamsted scientist on Newsnight saying something like, ’we’d welcome a dialogue so that we can explain what we’re going to do’ – that’s not really a dialogue, is it? More generally, if you feel very strongly about something, and you feel that you have been denied a legitimate voice, a common response is to protest. People who study conflicts know this very well.
Another useful exercise is to think if there is any (ostensibly legitimate, publicly funded) research that you personally would find so abhorrent that you’d be prepared to break the law in order to stop it. If so, would the researchers explaining their motives to you really be sufficient for you to stop protesting? I’m convinced that a lack of empathy is behind many of the world’s conflicts. I’m not defending the actions of the protestors, just trying to understand them. This is not science / anti-science, it’s politics.
(I’m GM agnostic too, by the way!)
I have a similar attitude. I’d like to support OA, but some of the proponents are just so arrogant, and even unpleasant. I also don’t think they understand how most scientists actually work, so they are proposing ideas which simply won’t be adopted. For example, they argue we shouldn’t have any pre-publication assessment of importance, but don’t tell us how we’re meant to sort through the fire-hose of new papers to find the ones which are of interest, and are likely to be important. Apparently we have to wait for post-publication metrics, even though nobody’s come up with a decent metric yet.
Thanks for the reply Tom, I like the thought exercises. I accept that we should have frank, open debates and for all parties to enter into these debates with humility, being willing to be persuaded by a sensible argument. Maybe some scientists involved in OA or GM debates haven’t been openminded enough to question or even see their own ideologies. But that still doesn’t justify extreme actions, nor does it justify a reluctance of agnostics to jump into the murky grey world of OA, GM, or whatever other issue floats your boat and challenge extremist views.
What if someone with a staunch objection to the computerisation of the modern world took it upon themselves to sabotage your analyses and delete all your data? If you had the chance before that person trashed your computer you would surely try and debate with them, to reason them out of it, and you would also call on nearby colleagues to take a stand as well.
Experiments are the fuel of science, necessary before you can even write a paper and decide whether to publish OA or not. Scientists should not surrender to protestors who abhor the questions behind an experiment, or the means by which the experiment is taking place, or the fact that public money is being used to pay for said experiment. Scientists, perhaps especially the agnostic scientists, should help defend these experiments. Maybe those who see an issue as black and white still won’t be swayed, but we should still try and engage with them.
Thanks Bob. One major implication of author pays is that it will place much more impetus on authors to select what they publish. In some ways this might be good (to the extent that there’s too much stuff out there) but faced with the choice of spending limited funds on publishing something high profile, or something worthy but dull, you’ll clearly go for the former. No more publication of negative results, in other words!
Charlie – I agree with a lot of what you say. I’m not justifying extreme actions, nor saying that there is some way of pleasing everyone. But equally, this means that if we work in contentious areas we should expect dissent and protests, and hope that the law will prevent their worst excesses. In the specific case of the GM trials, the offers of engagement only really came after the threat of protest, i.e. after the experiments had been planned/approved. It was never really going to work coming so late, and I can’t see that signing a petition is going to alter that (it’s the megaphone equivalent).
Right. Weekend time! Thanks for the thoughtful and thought-provoking comments.
Tom is on to something in comparing debates about open acces to debates about GM foods. He just got the sides wrong.
The GM debate pits people who are trying to apply science and human ingenuity to bring an antiquated and deeply stressed system (agriculture) into the 21st century and to make it more efficient and aligned to the needs of the public against anti-progress Luddites clinging to nostalgic views of the good old days.
Similarly, the OA debate pits people who are working to bring an antiquated and dysfunctional system (scientific publishing) into the 21st century and make it more efficient and aligned to the needs of the research community against anti-progress Luddites clinging to nostalgic views of the good old days.
A few points before you settle into your weekend:
(1) this
appears to assume maintenance of the journal funding status quo. Whatever your (or my) feelings about the whole OA discussion might be, to imagine that journal funding arrangements will continue as they have (overheads taken from grants to partly finance academic libraries who then pay publishers’ subscription fees) would be wrong. Many institutions already have deals with OA publishers, so individual members of those institutions don’t have to pay individual charges for each accepted article.
Alternative financial arrangements are easy to come up with, for either traditional OA, or the hybrid options that the new market is creating in traditional ‘for-profit’ publishers. If Sheffield doesn’t have such options in place, you need to get your megaphone out.
(2) The Journal of Negative Results – Ecology & Evolutionary Biology is completely OA and does not charge author fees. Two birds, one stone.
(3) Isn’t a blog just a megaphone with a global reach? How annoying must that be?
I appreciate that there are some rather strident voices in the pro-OA camp and parts of that Guardian editorial even made me wince but I think it’s worth bearing in mind that in any protest movement there are likely to be diverse voices. Since I have started writing in earnest about this issue (and informing myself along the way), I hope I have tried to maintain a degree of even-handedness and to examine some of the very real challenges facing the OA movement and that the OA movement poses to traditional publishers and to working academics in different fields.
Loud voices attract a lot of attention — and perhaps serve some purpose in drawing attention to the issue — but it would be unfortunate if they were to be given all the attention or come to be seen as the only representatives of the push for OA.
Ultimately there are many players in this game — scientists, funders, politicians and publishers — and it is good that they have all had a chance to weigh into the debate. The Guardian has done a particularly good job of providing an open platform for discussion. I participated in the #OADebate at Imperial College yesterday, which also involved representatives from all sides and I thought it was very constructive. There were differences of course, but we were able to have an intelligent debate about it. It may surprise some but there was also widespread agreement that OA is coming — everything is moving in that direction. The debate is largely now about how best to make it happen and how to manage the transition. In the UK that debate will no doubt continue when the Finch committee (which has had detailed discussions) reports at the end of June.
It is certainly a disruptive time — no-one is quite sure how things will play out. But I am glad that this issue has been brought to new prominence, even if that was through the initial very negative reaction to Elsevier’s actions of the Research Works Act. I am glad that the system is being perturbed. I think the scientific community has been too sleepy on this issue for too long; having woken up, we need to continue to think beyond traditional publishing models and to put publishers, many of whom are moving in the right direction, under continuing pressure to meet our needs.
I think you underestimate the problem of access, even in the relatively wealthy UK (see p43 of this publisher’s report (PDF). I share your concerns about the cost of publishing and see 35% profit margins as an unacceptable drain on public funds for science; part of my support for OA derives from the hope that, by making prices more visible (they are largely hidden from academics at the moment), this will stimulate market efficiencies that drive prices down.
And so, part of the process of perturbing the system so that we might shake up academic publishing, I have signed the White house petition. It won’t solve much on its own but it is a positive part of a very interesting dynamic.
Bob wrote: “(They) don’t tell us how we’re meant to sort through the fire-hose of new papers to find the ones which are of interest, and are likely to be important.”
I think it has never been easier to find papers that are interesting and important. Keyword searches work fine, and dig deeper into the literature than I ever could by visiting the library.
I know that some fields produce far more papers than others, and that people have many peripheral interests and want to read more widely than just papers on their own subject. Nevertheless, finding the most relevant papers for my own work is essentially a solved problem for me.
Yes, if you’re searching for a particular topic then I agree, it is much easier. But if you’re trying to to follow what’s going on in a subject, it’s not going to work: imagine searching for “ecology” every week. The only way I can see a keyword search working in practice is if you stick to a narrow focus (e.g. “species distribution models”). But this encourages specialisation: it makes it much more difficult to follow what’s going on outside a narrow area because you don’t have a pre-filter to tell you which papers aren’t that important.
Hi Tom, great post.
I’m currently involved in building up some infrastructure for open access publishing, though the creation of the journal eLife (http://www.elifesciences.org/).
We are setting out to try to prove that move value can be created within the scientific ecosystem through open access publishing, where that value is realised by authors readers and funders.
I personally believe that only through demonstrating this value can open access hope to be a model that comes to displace current models. The argument for readers is clear, immediate access, but the argument for authors is not so clear when career advancement is still tied strongly to the impact factor of a journal in many disciplines, especially for early career researchers at the post doctoral level.
We hope to help make that value clearer through an editorial policy that will publish the best research, and through developing article level metrics that we hope can make individual contributions easy to reward and not depend on a system that measures your worth based on a non scientific measure – the impact factor.
Ultimately though, this has to be about efficacy and not polemics. It has to be about finding a model that most efficiently reveals the the truths of the world around us, and that harnesses the web to amplify the knowledge that we can create as a species.
Oh, hi, ghost of Ian.
There’s a lot of talk about altmetrics, but has anyone come up with something any better than the IF?
Hey, if Roger Whittaker isn’t on board, then neither am I.
Right then (cracks knuckles)…
First, damn my grandma for picking this weekend to turn 90, I’ve missed all the fun here and on Twitter. During the course of which my point appears first to have been lost, and then to have resurfaced in a parallel ‘opposite’ world. All thanks to a throwaway line about GM.
So – I’m going to tackle these as I read them, in the order in which they appear, beginning with Michael. It seems important to start by quoting the full extent of what I said on the subject of GM: “My beef is not at all with OA, but rather in the way that the debate has been framed in terms of good and evil, right and wrong (not a million miles from the ongoing GM debate).” On Twitter this became an assertion that I was comparing OA advocates to GM destroyers, which is to wilfully miss the point.
My point actually was that polarising debate into science/anti-science or if you like Michael, progress/anti-progress is generally not helpful (and neither is bandying around names like ‘Luddite’). Most people are pro the science which aligns with their political ideology, and anti that which doesn’t. I come at this from an interest in the study conflicts (especially in a conservation context, legacy of a post-doc some years ago), which left me with a healthy appreciation of shades of grey. I thought this by Sunny Hundal in the Guardian was interesting on that front: http://www.guardian.co.uk/commentisfree/2012/may/25/greens-science-gm-food
But while my training inclines me to draw parallels between different situations, I don’t think you’ll find anything above in which I make any kind of equivalence between OA adherents and anti-GM protestors (unless we’re still in opposites world, in which case maybe you’re thinking of: “in no way do I want to equate pro-OA with anti-GM”). Yes, the post was a complaint about over-zealous campaigning turning me off. That’s my personal standpoint – I am constitutionally wary of conviction, certain only of my uncertainty, open (I hope) to reasoned debate but less so to megaphones. I specifically wrote about the more interesting debates, and acknowledged that plenty of OA supporters are clever, thoughtful and measured. Surely you wouldn’d deny that there are some zealots (in the sense of fervent supporters) too?
Regarding the notion of progress, it’s all to do with priorities. My personal view is that progress in science requires many things, and the reform of the way that we publish our primary research is some way down the list (still on the list, though, I hope you’ll note). Ahead of it I’d place the way we fund science, the way we assess and reward individuals and institutions, the way we attract, nurture and retain the best talent from right across society, and (crucially) the way we communicate our work in such a way as to be as effective as possible in making the world a better place – which is unlikely simply to involve making research papers more accessible. If I must be dragged into commenting on agriculture, I happen to think that GM will play an important role in feeding the world one day, but that there are more pressing priorities right now – we produce enough food for now, we just don’t distribute it equitably.
Mike – re Funding models, I’m interested to see what develops. But I know wiser people than me who are concerned that, whatever agreement Universities (and other research organisations) come to with OA publishers, there may be some kind of internal competition introduced which could lead to ‘rationing’ of publications between active researchers. Like I say, I find the subtleties of the debate, which go on below the shouting level, very interesting.
Re. -ve results, I set them up, you knock them down!
Re. megaphones, I said nobody’s ever persuaded by a megaphone. I reckon plenty of people are persuaded by books. A blog sits somewhere on that spectrum, I wouldn’t like to guess where…
Stephen – many thanks for your comment, I hope you realise that I had you down as one of the ‘clever and thoughtful’ people! I am certainly open to conversion, once the volume and shrillness of certain cheerleaders abates…
Zen – Bob has (as he has been doing over the weekend!) answered for me. If you like to keep tabs on a broad subject area, then the filtering service provided by journals is still very useful.
Ian – thanks for the comment, I’ve been sort-of aware of all the eLife stuff, should read up on what you’re up to more thoroughly. But I absolutely agree with your last point – it’s about efficacy. Which means defining (with some precision) where we want to get to, before we work out how to get there. I like the sound of your idea of our collective destination! (See, I’m NOT anti-progress after all!)
Happy birthday grandma. 90 is impressive.
The reason your post earned my ire was not the GM comparison. Rather it was the way that you parachuted into an issue you clearly have not studied carefully (hence the series of errors in your post) and smugly labeled its advocates as zealots whose views are not worthy of your attention.
This is a debate that has been going on for a long time, with the calls for boycotts and mandates and the what not possessing a long and fairly nuances history that you chose to completely ignore, or with which you are simply unfamiliar. (It might have helped if you’d pointed out where these shrill voices are, and what the “well-resourced single-issue campaign groups which will hear no alternative views” behind open access is. Are you talking about PLoS? If not, then who?)
The reason why I and so many other OA advocates favor funder mandates is not to dictate to individual scientists how they should publish, but rather to get funders and research institutions engaged in the issues that the transition to OA raises and which concern you. Only if they actually tackle things like how to shift payments from subscriptions managed by libraries to OA will it be possible to have what you want – OA without undermining existing journals.
You seem to equate open access with the destruction of journals, and trot out the old trope that open access seeks to get something that costs money for free. This is completely untrue. Open access has always been about changing the way we pay for publishing – not eliminating the payment. The idea is simple – take the money (~$10b/year) we as a research community spend on journals and re-channel it from subscriptions, which necessitate limited access, to up front payments, which allow unlimited free access and use.
Indeed the most prominent open access journal – PLoS Biology – was created almost a decade ago precisely to address the concern you have about maintaining the editorial role of journals in an open access world. And its success, along with several high-editorial touch OA journals – and now the launch of eLife – demonstrate that the equation you make of subscription journals and editorial filtering is inaccurate.
Let me also say that I agree with you that there are many other important issues in science – such as how we allocated funding, how individual scientists are assessed and how careers are built. But rather than seeing these as conflicting priorities, I see them all as integrally linked. The research community has, in many ways, punted its responsibility for figuring out who should get jobs, tenure, funding, etc… to journals. And I believe that a push by scientists to redefine what purpose scholarly communication serves and how they want it to be structured – including, but no exclusively, universal access to the output – would inevitably lead to progress in the many challenges vexing science today. I hope that even an “open access agnostic” could sign on to this.
Happy birthday grandma. 90 is impressive.
The reason your post earned my ire was not the GM comparison. Rather it was the way that you parachuted into an issue you clearly have not studied carefully (hence the series of errors in your post) and smugly labeled its advocates as zealots whose views are not worthy of your attention.
This is a debate that has been going on for a long time, with the calls for boycotts and mandates and the what not possessing a long and fairly nuances history that you chose to completely ignore, or with which you are simply unfamiliar. (It might have helped if you’d pointed out where these shrill voices are, and what the “well-resourced single-issue campaign groups which will hear no alternative views” behind open access is. Are you talking about PLoS? If not, then who?)
The reason why I and so many other OA advocates favor funder mandates is not to dictate to individual scientists how they should publish, but rather to get funders and research institutions engaged in the issues that the transition to OA raises and which concern you. Only if they actually tackle things like how to shift payments from subscriptions managed by libraries to OA will it be possible to have what you want – OA without undermining existing journals.
You seem to equate open access with the destruction of journals, and trot out the old trope that open access seeks to get something that costs money for free. This is completely untrue. Open access has always been about changing the way we pay for publishing – not eliminating the payment. The idea is simple – take the money (~$10b/year) we as a research community spend on journals and re-channel it from subscriptions, which necessitate limited access, to up front payments, which allow unlimited free access and use.
Indeed the most prominent open access journal – PLoS Biology – was created almost a decade ago precisely to address the concern you have about maintaining the editorial role of journals in an open access world. And its success, along with several high-editorial touch OA journals – and now the launch of eLife – demonstrate that the equation you make of subscription journals and editorial filtering is inaccurate.
Let me also say that I agree with you that there are many other important issues in science – such as how we allocated funding, how individual scientists are assessed and how careers are built. But rather than seeing these as conflicting priorities, I see them all as integrally linked. The research community has, in many ways, punted its responsibility for figuring out who should get jobs, tenure, funding, etc… to journals. And I believe that a push by scientists to redefine what purpose scholarly communication serves and how they want it to be structured – including, but no exclusively, universal access to the output – would inevitably lead to progress in the many challenges vexing science today. I hope that even an “open access agnostic” could sign on to this.
Michael – many thanks for the detailed comment, which is exactly the kind of thing which is likely to convert me, rather than aggressive Twitter misrepresentation. I’m sorry that you found my post to be smug, and I hope that the penultimate paragraph makes it clear that I am certainly not labelling all OA advocates as zealots. I think Stephen puts it well in his comment, that “in any protest movement there are likely to be diverse voices” and that gradually the substance replaces the bluster.
It’s important to note too that what I wrote about was a perception – and from comments here and elsewhere, it is not unique to myself – that some in the OA movement are not especially open to nuanced debate. So that every new petition or boycott becomes a ‘with us or against us’ moment. Of course questions of substance have been well considered in the OA and wider scientific community, and I apologise if I gave the impression that this was not the case. The rise of PLoS, one of the more notable developments in scholarly activity in recent years, could not have happened without substance underpinning it, and policy debates about OA would not be so well advanced. I’m delighted that these things are on the agenda.
Re. journals and costs – I don’t think I mentioned the destruction of journals. The ramifications of the economics of a switch to OA are fascinating – of course OA is not free, that was kind of my point! But so much of the economic rhetoric surrounding OA has been on the ‘don’t pay twice’, ‘publishers make x profit’ type, that the perception (that word again) that I get is that some see a dichotomy between ‘nasty expensive old publishers’ vs. ‘lovely free OA’. On editorial filtering – that was a point made by Bob in the comments, regarding post-publication review (rather than pre-publication review), so a different issue to OA. No-one, I think, would argue with the fact that some OA journals are editorially as rigorous as any (and so perfect filters from a reader’s perspective).
I’m going to risk drawing another parallel. The European Common Fisheries Policy is currently being reformed, and there’s been a really popular campaign to ensure that a discard (dumping fish at sea) ban is part of this reform. Most people (including me) agree that such a ban is highly desirable, but there is a concern among some scientists and policy experts that a discard ban may detract from some of the more fundamental reforms that are needed to ensure the fisheries remain viable. The fear is that politicians could get a lot of public support by bowing to pressure on this one issue, and not take steps to address underlying issues (like catching too many fish in the first place). I really hope that your vision of OA triggering change in other aspects of science proves to be correct, and that OA isn’t just something that politicians can sign up to, and get a lot of good PR, whilst burying plans for more substantial reform.
Finally, I bow to your superior knowledge of the issues I’ve written about. But what I’ve identified (and a few commenters have agreed with) is an image problem, which some of us find off-putting. Do with that information as you will.
Bob wrote: "There’s a lot of talk about altmetrics, but has anyone come up with something any better than the IF? "
Hmm, I wonder what data you have to back that up. I recently had a look at the data (you’ve seen our manuscript) and they tell me that it’s hard to do any worse than IF: IF predicts lower sample size and higher effect-size overestimate, IF doesn’t predict reproducibility, IF doesn’t predict sound methodology and IF predicts high retraction rates. If you have any data contradicting these studies, I’d like to cite them in our manuscript.
On top of all that, IF is negotiated, irreproducible and statistically unsound (which is public knowledge).
In other words. I can rank journals in any way I want and the data would probably look better than with an IF-based hierarchy. More to the topic, if you only read hi-IF journals, chances are, you’re missing out on all the reliable research that’s actually being published elsewhere.
You’ve seen our manuscript and posted a comment that was actually addressed six lines down. If after seeing our manuscript you still think IF is good, I’d be really interested in the data you have, because we must have missed it. We could not find any data backing up an IF-based hierarchy and would like to cite any such data.
Björn – I think you’re demonstrating Tom’s point about image. You don’t answer my question and instead attack the IF. Hey, we all know it’s crap (even those who – like me – didn’t read through all of your paper. Sorry, but I got distracted by other stuff).
I’m still interested in an answer – has anyone shown they’ve got anything better than the IF? At the moment it’s not obvious to me how altmetrics are going to describe scientific impact – in fact it’s not clear how impact is even defined operationally. Additionally, many of the problems with the IF will have to be addressed with other metrics (e.g. the time frame used, variation between research areas, gaming etc.).
Bob, I hope you pursue your research with a different attitude than you do discussions about scientific publishing. You admit that impact factor is crap, but rather than participate in a discussion at how to fix it, you complain that nobody has done it for you yet. You seem to find plenty of time to attack people who are working to make scientific communication better, but can’t be bothered to read something one of them sent trying to address a concern you explicitly raised?
You treat reforming the system for evaluating scientists and works of science as somehow independent of the existing system of scholarly communication. But surely you can see that a system in which people feel compelled to publish in journals with high IFs and libraries feel compelled to subscribe to these journals creates a situation in which reform of either how we evaluate science and how we pay for journals is needlessly difficult. The reason I and others push for things like funder/institutional mandates is to break this cycle.
It’s obviously your right not to care about these issues. But to claim you do, and then dismiss efforts to do something about it because you don’t like people’s tone is weak.
There are a lot of people working on altmetrics. I don’t think it’s unreasonable to ask if they’ve made any progress. This is the obvious question to ask about the altmetrics research agenda, and i don’t see why one shouldn’t be allowed to ask about the success of a research programme when sat on the outside.
FWIW, I have participated (in an small way, yes). Altmetrics isn’t my main area of work, which is why I didn’t follow it up (I would have had to spend a lot of time reading into counting processes first, which would have been difficult to justify spending my time on then and now). I guess I could email Martin and see if PLoS have any resources to throw at this.
Do I, really? How do you come to that conclusion? Actually, I don’t think so – it’s obvious that they’re connected (read the section on credit in Laboratory Life.
I don’t think I’ve dismissed anyone’s efforts to reform the system. I’ve dismissed Björn’s response to my question because, like yours, it didn’t answer my question.
Bob, we’re all a bit hot under the colar here. My point is that it’s not reasonable to condition the move to open access on the existence of altmetrics to replace IF, in part because open access doesn’t obligately undermine IFs and the existing journal hierarchy, but also because I see open access as an essential first step to actually improving and replacing it.
There’s also an interesting conversation to have about IF and altmetrics. I actually think IF is deeply misunderstood, and in some ways underappreciated. While it is inarguably a horrible statistic (a mean of a distribution that is not even vaguely normal) and is manipulated and misused in all manner of ways (especially in the ways journals chase IF with sexy/glamour articles at the expense of quality). But it – or more precisely the hierarchy of journals arranged on the basis of IF – nonetheless carries valuable information, in that it is the only way we currently have for reviewers and editors to record their prediction of how important and useful a work will ultimately be.
The problem, to me, is that a collection of thousands of different journal titles is a really clumsy and stupid way to record this assessment. What I want to see from altmetrics is not just better ways to measure how much of an impact a work is having – though this is important work – but also better ways for scientists to record how important they think a work will be and to whom it will be important. If we can do this, the impetus to keep the existing collection of journals, and the legacy of the subscription-based business model they carry, will disappear.
I’ve written on this before, but it’s an issue that warrants a lot more discussion, creativity, and, most importantly, work.
http://www.michaeleisen.org/blog/?p=694
Bob, this is confusing, I’m sure we can clear this up.
You asked for “anything better than IF” and I answered by citing data that suggest that “it’s hard to do any worse than IF”, meaning that indeed anything IS better than IF.
Just to be sure this wasn’t missed, I added: “I can rank journals in any way I want and the data would probably look better than with an IF-based hierarchy.” Thus, I think I answered your question twice: first I said that anything is better than IF (you asked for anything better than IF), then I said that I (meaning, of course, anybody) could come up with any other arbitrary ranking of journals and do better than IF.
Thus, if you wanted to use journal-based filtering/ranking, you could either randomize the journals yourself or pick any of the other journal ranks: 5-year IF, Google’s h-index (highly recommended!), Eigenfactor, SciMago Journal Rank, etc. My understanding of the data is that you’re very likely to do better than using IF.
I can only guess that maybe you didn’t take me seriously and hence thought I was “demonstrating Tom’s point about image”. However, I was using data to very seriously answer your question. My answer is, just to make it unambiguously clear, that the data I have read lead me to believe that indeed “anything is better than IF”.
Just to add another reference to the 80-something in our manuscript supporting our conclusions:
“The weakening relationship between the Impact Factor and papers’ citations in the digital age”
http://arxiv.org/abs/1205.4328
It seems like fewer and fewer people are paying attention to IF and also that shows in the data. Even at the height of IF, the coefficient of determination has never been higher than a measly 0.3, with a guesstimate of a median of below 0.2.
With the wealth of data suggesting an at least equally if not better predictive power of IF with low sample size, lack of experimental rigor or retractions, etc., any score that eliminates all correlations (e.g. a random ranking) will do better than IF simple by eliminating all the ‘bad’ correlations. Thus, quite literally, the data seem to suggest that anything is better than IF.
I hope this is even-handed, data-centric and answers your question. I’m not being sarcastic, facetious or ironic. I mean and meant exactly what I wrote. Of course, as always, I’ll gladly reverse my position to the opposite if there is better data around that I’m not aware of.
Just read Bob’s and Michael’s posts. Clearly, using a randomized list of journals is not what we’re aiming for, of course. As scientists, I’m sure we can do much better than that. However, that wasn’t Bob’s question.
IMHO, altmetrics today are already better than any journal-based sorting/ranking (see Jason Priem’s and Heather Piwovar’s work) – what’s lacking is a proper implementation into a scientist’s workflow.
I agree. I’ve interpreted the OA movement’s focus on altmetrics to largely be tactical – because the OA finances works poorly with journals that are very selective, reducing the importance of journal selectivity helps OA. And this shouldn#t be seen as a criticism (and I also know that this isn’t the only reason for the interest in altmetrics, and for many altmetricians it may not be their main motivation).
The problem, and this may largely be a presentational one, is that there doesn’t seem to have been much of a shift in the rhetoric beyond “The IF is bad but altmetrics will save us”. I haven’t seen much movement beyond that, hence my scepticism. Things might move beyond this, and I hope it does. But at the moment all altmetrics is giving us is a promissory note.
Actually, I have a deeper scepticism that might not be allayed: altmetrics is meant to find other ways of measuring “impact”, but how does it define “impact”? Obviously this is a complex question, but unless you have an answer – and an answer that can be operationalised – the whole programme looks in trouble. This is a philosophical problem with a practical sting – if you can’t define impact, what do you benchmark your metrics against? IOW, how do you find out how well they’re estimating impact?
A couple of comments:
1. Why do you consider lack of normality such a problem? Obviously the citations are positively skewed, but the number of papers produced by a journal is large enough that the central limit theorem should kick in nicely, so the IF will be pretty Gaussian.
2. Some of the problems with the IF could be alleviated using a more complex scheme weighting scheme – rather than give everything in the last 2 years equal weight, make the weight reduce the further in the past you go. The rate of weight change would have to be decided, but I think that can be done adaptively, so that it optimises the predictive performance. But more work etc. etc. And it’ll probably still be crap.
Yes, true. But journals also provide a short-hand for other things: the subject area and (perceived) merit of the work. The former is useful for deciding which eTOCs to subscribe to (as I argued above, I’m not convinced keywords will work for browsing new papers). The latter is obviously used as a marker for assigning credit. This is obviously what you want altmetrics to replace, but I’m not sure they’ll be accepted as the way to do it (largely because you have to wait for them to tell you anything). But that’s a much bigger discussion.
Bob O’Hara asked: “There’s a lot of talk about altmetrics, but has anyone come up with something any better than the IF?”
Yes. I propose ranking journals in alphabetical order of title. Doing so will at least not yield a positive correlation between rank and retraction rate, so it will be a better ranking than impact factor.
Also: I want to see people saying, “Yeah, we tried to get into Acta Palaeontological Polonica, but we were rejected without review, and had to work our way down through BMC Biology, Cretaceous Research, and so on, all the way down to the Zoological Journal of the Linnean Society”.
Yet another advantage to my proposed new journal-ranking scheme: it will make Nature a mid-ranked journal.
[...] Open Access and further still many academics are notably apathetic towards it, or are even proudly agnostic on the [...]
'Proudly agnostic'? Jeez. Wish people would (a) read what I wrote, (b) look up words they don't understand