What Lies Beneath – The Hidden World of Clinical Trials

I know a few scientists from some quite varied disciplines, and one or two friends are climate scientists. I don't know a great deal about climate science and what they do all day, but in my imagination they spend a lot of time on sturdy ships contemplating icebergs with a flask of tea to hand, rather than plotting variables on a climate modelling program in an office somewhere - I can't help it, I'm the romantic type.

The thing about icebergs is, they really are incredibly big. There's the bit above the water line, which everyone can see, but the majority of this massive floating chunk is actually invisible below the surface. It's what any 19th century sea dog was truly afraid of - not what was on top, it was visible and relatively easy to steer around. It was the invisible stuff below that might scrape your hull to bits that you worried about.

In my line (clinical epidemiology is my especial interest) you get entirely another kind of iceberg, and it's made of data. The small amount that we can glean from journals and trial registries is there for all to see, but the vast majority is hidden away and we're trying to navigate the waters of medical knowledge without knowing what's under there. Our funnel plots and other strategies for detecting publication bias help, but it's the equivalent of the young lad in the crow's nest at the top of the mast. We need more. It's the 21st century. We need maps, and sonar.

Big Business, Big Investments, Big Data

The modern pharmaceutical industry has one enormous advantage in this respect. They're required, because it makes good business sense, to draw those maps.  They're called Clinical Study Reports, and they are the vast underside of our iceberg. You wouldn't make a multi-million pound investment in a new treatment without a damn good idea it will work. The average clinical trial produces thousands of pages of paperwork, from enormous protocols (detailed plans of how the trial will take place) through to oceans of raw data at the end. The only problem is, most CSRs are never made publicly available due to claims of commercial sensitivity or patient data protection. A vanguard of campaigners are politely pointing out that these excuses can be overcome if we really want to, and a host of court cases and Freedom of Information requests are proving it.

Doctors have the right to see this information to make fully informed decisions, the public have the right to know if what they're spending money on really is their best shot or just a bit of spin, and the patients who took part in those trials have a right to see some result come from their efforts. In fact, that last is enshrined in the Declaration of Helsinki, the  statement of The World Medical Association's principles regarding clinical trials using human subjects. All trials registered, all trials reported.

Yet that declaration was made almost 50 years ago, in 1964. We're still here. Too many trials are abandoned. Too many never see publication, and even more are subject to outcome reporting bias - reporting some of the results, but not all of them - to spin a more favourable result.

All Trials

In January, former Guardian columnist and Bad Science blogger Ben Goldacre collaborated with the charity Sense About Science to herald the All Trials campaign, and so far they're doing rather well. Numerous professional organisations, patient groups and even some individual pharmaceutical companies have signed up to the All Trials cause and strongly signalled their commitment to more openness. There seems to be a tremendous level of energy and commitment to the campaign which, to me, comes from the fact this is the first time the situation has been explained to the "patients" whom this secrecy affects - the general public.

So what do All Trials suggest? The latest All Trials document, published a few weeks ago, explains the 4 levels of reporting that could convey information about a trial to the public.


All Trials would like, well.... all trials to be registered. Easy, right? In fact it's now a legal requirement, so what's the problem? Well, according to All Trials this does nothing for the trials that took place before the legal requirement to register them - and these trials cover the huge majority of everyday medicines in use today. The stuff you take for your dicky ticker, the high blood pressure, the asthma treatments and  anti-depressants? Practically the only information we know for some common treatments is that a trial must have taken place at some point. How well or badly it was conducted, what the result was, who knows? This is why All Trials would also like retrospective registration for trials that look place before registration was a legal requirement.

The World Health Organisation has a list of 20 pieces of information that should be the minimum for registering trials, including the protocol, information about participants and their informed consent, the intervention being investigated, the primary outcomes which will be measured, a data collection plan and a statistical analysis plan. All Trials point out that many trial registrations are incomplete with relation to the WHO list and should be completed.

Summary Results Reporting

All Trials would like to see a summary of the results published within a year of the trial being completed. In January 2012, Ross et al reported to the BMJ that of 635 trials registered on clinicaltrials.gov and completed by December 2008, only 20% produced a summary within a year - already a legal requirement in the US, but a somewhat toothless one as fines for lack of compliance are both rare and insignificant compared to the vast income of the companies involved. On a more optimistic note, 46% were peer reviewed and published on Medline within 30 months, but the median average was 51 months for results to see the light of day - over 4 years. And 32% of summary results had not been published at all to date.

All Trials campaign for the legislation to be observed and mirrored by Europe - summary results within a year, with a list of minimum requirements including the protocol and any amendments that were made to it, the participant information, the results of the primary outcome and any key secondary outcomes, the statistical analysis and reporting of any adverse effects. Ideally the results should be searchable, and sharable between the formats of different Trial Registry set-ups.

Full Reports

All Trials suggest that where companies are producing full reports anyway (which they need to for marketing approval) then they should make these public. The campaign admits that the whole area of individual patient data is incredibly tricky, and while others are looking into ways that anonymised data can be made available All Trials do not see this issue as within their remit. Instead they suggest that information on both individual patients  and adverse effects (which may make someone identifiable) be redacted from public record, but made available to researchers on request.

It may be surprising to know that medical funding groups and patient groups are actually very keen for individual patient data to be made available, as long as privacy concerns can be dealt with. How much better for researchers wanting to replicate or seek a new angle for a subset of the population if they have access to that data, rather than to have to start from scratch?


There's no doubt from their support that the medical community considers All Trails a worthy cause to get behind. Trying to make medical decisions without all the information being available to you is a bit like sailing your ship through icy water. Imagine you had a passenger who knew what was under you, who could give you detailed maps and help you navigate. But instead they hid their maps away in their cabin and stood next to you, telling you how beautiful and marvellous the icebergs you could see are. You ask them for the map to navigate but they become vague. They're not sure if they have it, they can tell you how big that one is over there but the shape is confidential so who knows if there's a bit that will bash a hole in your ship? It would be frustrating to say the least. You'd be looking to your crew to go to the passenger's cabin and find that map. And one group of researchers has recently suggested that we do just that. If the information from Clinical Study Reports is hidden, we should simply go and take it.

Restoration Man

In June of this year, Peter Doshi and colleagues wrote about a bold scheme in the BMJ - quite simply publish, or we'll do it for you. The plan was called RIAT - Restoring Invisible and Abandoned Trials, and the suggested method was so simple it was inspired. Doshi et al invite all funders or trial conductors to put unpublished information from Clinical Study Reports out there within a year or else, using Freedom of Information requests and the information made available from litigation, a team of volunteer researchers  world wide would seek it out themselves and publish summaries. As Doshi et al put it so succinctly:

"Because abandonment can lead to false conclusions about effectiveness and safety, we believe that it should be tackled through independent publication and replication of trials"

Of course there is still a risk of outcome reporting bias - the average journal article or summary of a study may compress the Clinical Study Report by a factor of 1000:1, and Doshi et al are clear there will be strict guidelines and transparency in place to show the reasoning behind what gets left out. The analysis was published in the BMJ together with an editorial from Fiona Godlee and Ginny Barbour of PLOS among others urging the move, encouraging other publications to sign up to the RIAT principle and encouraging discussion on the details.

Of course, medicine and healthcare fields are not the inventors of Big Data and Freedom of Information, it's a movement that's been gaining pace in all kinds of applications in the last few years and is changing journalism and activism for good. But I'm especially excited about where these campaigns might lead us with clinical trials. I believe that the next big revolution in health won't be a new treatment or discovery, but the really big issue of how we use the information that already exists. It's out there. We just need to politely insist that we'd like the maps.

4 Responses to “What Lies Beneath – The Hidden World of Clinical Trials”

  1. Adam Jacobs Reply | Permalink

    RIAT is a nice idea. Anything that gets unpublished trials into the public domain is welcome.

    However, there is a huge practical problem with it, which I've not seen any of its proponents even mention, let alone try to propose a solution to it.

    That problem is cost. Papers don't write themselves. Particularly for long completed studies where details can be hard to track down, there is a huge amount of work involved. Who is going to pay for it?

    While I like the idea of RIAT, I suspect it's never going to get off the ground until someone starts to think about an answer to that question.

  2. Paige Brown Jarreau Reply | Permalink

    Great article, important subject. You make a clear case for openness if clinical trial reports.

    I've also been looking for potential science research charities that people can donate too. What do you think about 'Sense about Science'?

  3. daniel Reply | Permalink

    I am also agree with you on these clinical trials....And I appreciate you that you came forward and took time to write about this..

  4. Michael Reply | Permalink

    "there is a huge amount of work involved. Who is going to pay for it?"
    I have contacted Dr Doshi to volunteer some of my time unpaid.

Leave a Reply

six + = 10