1. Novelty of the scientific research:

Most researchers routinely claim that their findings are novel, but are the claims of novelty appropriate? Is the research pointing towards a fundamentally new biological mechanism or introducing a completely new scientific tool? Or does it just represent a minor incremental growth in our understanding of a biological problem?

2. Significance of the research:

How does the significance of the research compare to the significance of other studies in the field? A biological study might uncover new regulators of cell death or cell growth, but how many other such regulators have been discovered in recent years? How does the magnitude of the effect in the study compare to magnitude of effects in other research studies? Suppressing a gene might prolong the survival of a cell or increase the regeneration of an organ, but have research groups published similar effects in studies which target other genes? Some research studies report effects that are statistically significant, but are they also biologically significant?

3. Replicability:

Have the findings of the scientific study been replicated by other research groups? Does the research study attempt to partially or fully replicate prior research? If the discussed study has not yet been replicated, is there any information available on the general replicability success rate in this area of research?

4. Experimental design:

Did the researchers use an appropriate experimental design for the current study by ensuring that they included adequate control groups and addressed potential confounding factors? Were the experimental models appropriate for the questions they asked and for the conclusions they are drawing? Did the researchers study the effects they observed at multiple time points or just at one single time point? Did they report the results of all the time points or did they just pick the time points they were interested in?

Examples of issues: 1) Stem cell studies in which human stem cells are transplanted into injured or diseased mice are often conducted with immune deficient mice to avoid rejection of the human cells. Some studies do not assess whether the immune deficiency itself impacted the injury or disease, which could be a confounding factor when interpreting the results. 2) Studies which investigate the impact of the 24-hour internal biological clock on the expression of genes sometimes perform the studies in humans and animals who maintain a regular sleep-wake schedule. This obscures the cause-effect relationship because one is unable to ascertain whether the observed effects are truly regulated by an internal biological clock or whether they merely reflect changes associated with being awake versus asleep.

5. Experimental methods:

Are the methods used in the research study accepted by other researchers? If the methods are completely novel, have they been appropriately validated? Are there any potential artifacts that could explain the findings? How did the findings in a dish ("in vitro") compare to the findings in an animal experiment ("in vivo")? If new genes were introduced into cells or into animals, was the level of activity comparable to levels found in nature or were the gene expression levels 10-, 100- or even 1000-fold higher than physiologic levels?

Examples of issues: In stem cell research, a major problem faced by researchers is how stem cells are defined, what constitutes cell differentiation and how the fate of stem cells is tracked. One common problem that has plagued peer-reviewed studies published in high-profile journals is the inadequate characterization of stem cells and function of mature cells derived from the stem cells. Another problem in the stem cell literature is the fact that stem cells are routinely labeled with fluorescent markers to help track their fate, but it is increasingly becoming apparent that unlabeled cells (i.e. non-stem cells) can emit a non-specific fluorescence that is quite similar to that of the labeled stem cells. If a study does not address such problems, some of its key conclusions may be flawed.

6. Statistical analysis:

Did the researchers use the appropriate statistical tests to test the validity of their results? Were the experiments adequately powered (have a sufficient sample size) to draw valid conclusions? Did the researchers pre-specify the number of repeat experiments, animals or humans in their experimental groups prior to conducting the studies? Did they modify the number of animals or human subjects in the experimental groups during the course of the study?

7. Consensus or dissent among scientists:

What do other scientists think about the published research? Do they agree with the novelty, significance and validity of the scientific findings as claimed by the authors of the published paper or do they have specific concerns in this regard?

8. Peer review process:

What were the major issues raised during the peer review process? How did the researchers address the concerns of the reviewers? Did any journals previously reject the study before it was accepted for publication?

9. Financial interests:

How was the study funded? Did the organization or corporation which funded the study have any say in how the study was designed, how the data was analyzed and what data was included in the publication? Do the researchers hold any relevant patents, own stock or receive other financial incentives from institutions or corporations that could benefit from this research?

10. Scientific misconduct, fraud or breach of ethics

Are there any allegations or concerns about scientific misconduct, fraud or breach of ethics in the context of the research study? If such concerns exist, what are the specific measures taken by the researchers, institutions or scientific journals to resolve the issues? Have members of the research team been previously investigated for scientific misconduct or fraud? Are there concerns about how informed consent was obtained from the human subjects?

This is just a preliminary list and I would welcome any feedback on how to improve this list in order to develop tools for assessing the critical analysis content in science writing. It may not always be possible to obtain the pertinent information. For example, since the peer review process is usually anonymous, it may be impossible for a science writer to find out details about what occurred during the peer review process if the researchers themselves refuse to comment on it.

One could assign a point value to each of the categories in this checklist and then score individual science news articles or science blog-posts that discuss specific research studies. A greater in-depth discussion of any issue should result in a greater point score for that category.

Points would not only be based on the number of issues raised but also on the quality of analysis provided in each category. Listing all the funding sources is not as helpful as providing an analysis of how the funding could have impacted the data interpretation. Similarly, if the science writer notices errors in the experimental design, it would be very helpful for the readers to understand whether these errors invalidate all major conclusions of the study or just some of its conclusions. Adding up all the points would then generate a comprehensive score that could become a quantifiable indicator of the degree of critical analysis contained in a science news article or blog-post.

********************

EDIT: The checklist now includes a new category - scientific misconduct, fraud or breach of ethics.

One of the themes that emerged in response to the article was the Us-vs.-Them perception that “scientists” were attacking “journalists”. This was surprising because as a science blogger, I assumed that I, too, was a science journalist. My definitions of scientist and journalist tend to be rather broad and inclusive. I think of scientists with a special interest and expertise in communicating science to a broad readership as science journalists. I also consider journalists with a significant interest and expertise in science as scientists. My inclusive definitions of scientists and journalists have been in part influenced by an article written by Bora Zivkovic, an outstanding science journalist and scientist and the person who inspired me to become a science blogger.  As Bora Zivokovic reminds us, scientists and journalists have a lot in common: They are supposed to be critical and skeptical, they obtain and analyze data and they communicate their findings to an audience after carefully evaluating their data.  However, it is apparent that some scientists and journalists are protective of their respective domains. Some scientists may not accept science journalists as fellow scientists unless they are part of an active science laboratory. Conversely, some journalists may not accept scientists as fellow journalists unless their primary employer is a media organization. For the purpose of this discussion, I will try to therefore use the more generic term “science writing” instead of “science journalism”.

Are infotainment science writing and critical science writing opposites? This was one of the major questions that arose in the Twitter discussion. The schematic below illustrates infotainment and critical science writing.

Although this schematic of a triangle might seem oversimplified, it is a tool that I use to help me in my own science writing. “Critical science writing” (base of the triangle) tends to provide information and critical analysis of scientific research to the readers. Infotainment science writing minimizes the critical analysis of the research and instead focuses on presenting content about scientific research in an entertaining style. Scientific satire as a combination of entertainment and critical analysis was not discussed in the Guardian article, but I think that this too is a form of science writing that should be encouraged.

Articles or blog-posts can fall anywhere within this triangle, which is why infotainment and critical science writing are not true dichotomies, they just have distinct emphases. Infotainment science writing can include some degree of critical analysis, and critical science writing can be somewhat entertaining. However, it is rare for science writing (or other forms of writing) to strike a balance that is able to include accurate scientific information, entertainment, as well as a profound critical analysis that challenges the scientific methodology or scientific establishment, all in one article. In American political journalism, Jon Stewart and the Daily Show are perhaps one example of how one can inform, entertain and be critical – all in one succinct package. Currently, contemporary science writing which is informative and entertaining (‘infotainment’), rarely challenges the scientific establishment the way Jon Stewart challenges the political establishment.

Is ‘infotainment’ a derogatory term?  Some readers of the Guardian article assumed that I was not only claiming that all science journalism is ‘infotainment’, but also putting down ‘infotainment’ science journalism. There is nothing wrong with writing about science in an informative and entertaining manner, therefore ‘infotainment’ science writing should not be construed as a derogatory term. There are differences between good and sloppy infotainment science writing. Good infotainment science writing is accurate in terms of the scientific information it conveys, whereas sloppy infotainment science writing discards scientific accuracy to maximize hype and entertainment value. Similarly, there is good and sloppy critical science writing. Good critical science writing is painstakingly careful in the analysis of the scientific data and its scientific context by reviewing numerous other related scientific studies in the field and putting the scientific work in perspective. Sloppy critical science writing, on the other hand, might just single out one scientific study and attempt to discredit a whole area of research without examining context. Examples of sloppy critical science writing can be found in the anti-global warming literature, which hones in on a few minor scientific discrepancies, but ignores the fact that 98-99% of climate scientists agree on the fact that humans are the primary cause of global warming.

Instead of just discussing these distinctions in abstract terms, I will use some of my prior blog-posts to illustrate differences between different types of science writing, such as infotainment, critical science writing or scientific satire. I find it easier to critique my own science writing than that of other science writers, probably because I am plagued by the same self-doubts that most writers struggle with. The following analysis may be helpful for other science writers who want to see where their articles and blog-posts fall on the information – critical analysis – entertainment spectrum.

A.     Infotainment science writing

Infotainment science writing allows me to write about exciting or unusual new discoveries in a fairly manageable amount of time, without having to extensively review the literature in the field or perform an in-depth analysis of the statistics and every figure in the study under discussion. After providing some background for the non-specialist reader, one can focus on faithfully reporting the data in the paper and the implications of the work without discussing all the major caveats and pitfalls in the published paper. This writing provides a bit of an escapist pleasure for me, because so much of my time as a scientist is spent performing a critical analysis of the experimental data acquired in my own laboratory or in-depth reviews of scientific manuscripts and grants of either collaborators or as a peer reviewer. Infotainment science writing is a reminder of the big picture, excitement and promise of science, even though it might gloss over certain important experimental flaws and caveats of scientific studies.

Infotainment Science Writing Example 1: Using Viagra To Burn Fat

This blog-post discusses a paper published in the FASEB Journal, which suggested that white (“bad”) fat cells could be converted into brown (“good”) fat cells using Viagra. The study reminded me of a collision between two groups of spam emails: weight loss meets Viagra. The blog-post provides background on white and brown adipose tissue and then describes the key findings of the paper. A few limitations of the study are mentioned, such as the fact that the researchers never document weight loss in the mice they treated, as well as the fact that the paper ignores long-term consequences of chronic Viagra treatment. The reason I consider this piece an infotainment style of science writing is that there were numerous criticisms of the research study that could have been brought to the attention of the readers. The researchers concluded the fat cells were being converted into brown fat using only indirect measures without adequately measuring the metabolic activity and energy expenditure. It is not clear why the researchers did not extend the duration of the animal studies to show that the Viagra treatment could induce weight loss. If all of these criticisms had been included in the blog-post, the fun Viagra-weight loss idea would have been drowned in a whirlpool of details.

Infotainment Science Writing Example 2: The Healing Power of Sweat Glands

The idea of “icky” sweat glands promoting wound healing was the main hook. Smelly apocrine sweat glands versus eccrine sweat glands are defined in the background of this blog-post, and the findings of the paper published in the American Journal of Pathology are summarized.  Limitations of the study included little investigation of the mechanism of regeneration, whether cells primarily proliferate or differentiate to promote the wound healing and an important question: Does sweating itself affect the regenerative capacity of the sweat glands? Although these limitations are briefly mentioned in the blog-post, they are not discussed in-depth and there is no comparison made between the observed wound healing effects of sweat gland cells to the wound healing capacity of other cells. This blog-post is heavy on the “information” end, and it provides little entertainment, other than evoking the image of a healing sweat gland.

B.     Critical science writing

Critical science writing is exceedingly difficult because it is time-consuming and challenging to present critiques of scientific studies in a jargon-free manner. An infotainment science blog-post can be written in a matter of a few hours. A critical science writing piece, on the other hand, requires an in-depth review of multiple studies in the field to better understand the limitations and strengths of each report.

Critical Science Writing Example 1: Bone Marrow Cell Infusions Do NOT Improve Cardiac Function After Heart Attack

This blog-post describes an important negative study conducted in Switzerland. Bone marrow cells were injected into the hearts of patients in one of the largest randomized cardiovascular cell therapy trials performed to date. The researchers found no benefit of the cell injections on cardiac function. This research has important implications because it could stave off quack medicine. Clinics in some countries offer “miracle cures” to cardiovascular patients, claiming that the stem cells in the bone marrow will heal their diseased hearts. Desperate patients, who fall for these scams, fly to other countries, undergo risky procedures and end up spending $20,000 or $40,000 out of pocket for treatments that simply do not work. This blog-post is in the critical science writing category because it not only mentions some limitations of the Swiss study, but also puts the clinical trial into context of the problems associated with unproven therapies. It does not specifically discuss other bone marrow injection studies, but it provides a link to an editorial I wrote for an academic journal which contains all the pertinent references. A number of readers of the Guardian article raised the question whether one can make such critical science writing appear entertaining, but I am not sure how to incorporate entertainment into this type of an analysis.

Critical Science Writing Example 2: Cellular Alchemy: Converting Fibroblasts Into Heart Cells

This blog-post was a review of multiple distinct studies on converting fibroblasts – either found in the skin or the hearts – into beating heart cells. The various research groups described the outcomes of their research, but the studies were not perfect replications of each other. For example, one study that reported a very low efficiency of fibroblast conversion not only used cells derived from older animals but also used a different virus to introduce the genes. The challenge for a critical science writer is to decide which of these differences need to be highlighted, because obviously not all differences and discrepancies can be adequately accommodated in a single article or blog-post. I decided to highlight the electrical heterogeneity of the generated cells as the major limitation of the research because this seemed like the most likely problem when trying to move this work forward into clinical therapies. Regenerating a damaged heart following a heart attack would be the ultimate goal, but do we really want to create islands of heart cells that have distinct electrical properties and could give rise to heart rhythm problems?

C.     Science Satire

In closing, I just want to briefly mention scientific satire – satirical or humorous descriptions of real-life science. One of the best science satire websites is PhD Comics, because the comics do a brilliant job of portraying real world science issues, such as the misery of PhD students and the vicious cycle of not having enough research funding to apply for research funding. My own attempts at scientific satire take the form of spoof news articles such as “Professor Hands Out "Erase Undesirable Data Points" Coupons To PhD Students” or “Academic Publisher Unveils New Journal Which Prevents All Access To Its Content”. Science satire is usually not informative, but it can provide entertainment and some critical introspection. This kind of satire is best suited for people with experiences that allow them to understand inside jokes. I hope that we will see more writing that satirizes the working world of how scientists interpret data, compete for tenure and grants or interact with graduate students.

The study entitled “Quantifying the consensus on anthropogenic global warming in the scientific literature” was published by John Cook and colleagues as an open access paper in the journal Environmental Research Letters. The results are no surprise to anyone who has been following the scientific literature on AGW. Of the abstracts that expressed an opinion on AGW, 97.1% explicitly or implicitly stated that humans were the primary cause of global warming. This high level of consensus on the primary human role in causing global warming is very consistent with prior publications in the field. When Cook and colleagues contacted the authors of the papers to obtain their own opinion on the matter, they found that 98% of the authors who had a clear position on climate change agreed on human activity being the major cause of global warming.

Even though I think that the conclusions of the study are correct and that there is indeed a 97-98% consensus among scientists on AGW, I feel that the study highlights the potential for bias in “citizen science”.

The idea of using “citizens”, i.e. volunteers who are not necessarily trained as scientists to help obtain data is very intriguing. However, I do not believe that the authors of the study adequately addressed the issue of potential bias among these volunteers. The paper mentions that they contributed to the website Skeptical Science, which is managed by John Cook and attempts to convert climate change skeptics, i.e. people who deny the primary role of humans in global warming. I suspect that the volunteers who contribute to the website are probably all strongly convinced that AGW is very real. This could introduce a bias in the grading of the abstracts by these volunteers. I could not find any part of the paper, which discussed this potential bias and whether the authors also considered using “citizens” as abstract evaluators who did not believe in AGW or volunteers who felt neutral about AGW. Such “citizens” would have been good control groups to test whether a pre-existing opinion among volunteers can bias their interpretation of the scientific literature.

These concerns about potential "citizen science" bias should not only be addressed in the context of global warming research, but also in other areas of science that are associated with controversy and strongly held beliefs. A “citizen science” assessment of the risks and benefits of gene therapy or of embryonic stem cells in the scientific literature might also be influenced by their beliefs. As excited as many of us are about “citizen science”, it is necessary for us to consider potential biases that “citizens” can introduce, just like we also take into account the biases of professional scientists who conduct experiments when we evaluate a scientific paper.

Image credit: Annual average global warming by the year 2060 simulated and plotted via NASA)

Fibroblasts are an abundant cell type, found in many organs such as the heart, liver and the skin. One of their main functions is to repair wounds and form scars in this process. They are fairly easy to grow or to expand, both in the body as well as in a culture dish. The easy access to large quantities of fibroblasts makes them analogous to the “base metals” of the alchemist. Adult cardiomyocytes, on the other hand, are not able to grow, which is why a heart attack which causes death of cardiomyocytes can be so devastating. There is a tiny fraction of regenerative stem-cell like cells in the heart that are activated after a heart attack and regenerate some cardiomyocytes, but most of the damaged and dying heart cells are replaced by a scar – formed by the fibroblasts in the heart. This scar keeps the heart intact so that the wall of the heart does not rupture, but it is unable to contract or beat, thus weakening the overall pump function of the heart. In a large heart attack, a substantial portion of cardiomycoytes are replaced with scar tissue, which can result in heart failure and heart failure.

A few years back, a research group at the Gladstone Institute of Cardiovascular Disease (University of California, San Francisco) headed by Deepak Srivastava pioneered a very interesting new approach to rescuing heart function after a heart attack.  In a 2010 paper published in the journal Cell, the researchers were able to show that plain-old fibroblasts from the heart or from the tail of a mouse could be converted into beating cardiomyocytes! The key to this cellular alchemy was the introduction of three genes – Gata4, Mef2C and Tbx5 also known as the GMT cocktail– into the fibroblasts. These genes encode for developmental cardiac transcription factors, i.e. proteins that regulate the expression of genes which direct the formation of heart cells. The basic idea was that by introducing these regulatory factors, they would act as switches that turn on the whole heart gene program machinery. Unlike the approach of the Nobel Prize laureate Shinya Yamanaka, who had developed a method to generate stem cells (induced pluripotent stem cells or iPSCs) from fibroblasts, Srivastava’s group bypassed the whole stem cell generation process and directly created heart cells from fibroblasts. In a follow-up paper published in the journal Nature in 2012, the Srivastava group took this research to the next level by introducing the GMT cocktail directly into the heart of mice and showing that this substantially improved heart function after a heart attack. Instead of merely forming scars, the fibroblasts in the heart were being converted into functional, beating heart cells – cellular alchemy with great promise for new cardiovascular therapies.

As exciting as these discoveries were, many researchers remained skeptical because the cardiac stem cell field has so often seen paradigm-shifting discoveries appear on the horizon, only to later on find out that they cannot be replicated by other laboratories. Fortunately, Eric Olson’s group at the University of Texas, Southwestern Medical Center also published a paper in Nature in 2012, independently confirming that cardiac fibroblasts could indeed be converted into cardiomyocytes. They added on a fourth factor to the GMT cocktail because it appeared to increase the success of conversion. Olson’s group was also able to confirm Srivastava’s finding that directly treating the mouse hearts with these genes helped convert cardiac fibroblasts into heart cells. They also noticed an interesting oddity. Their success of creating heart cells from fibroblasts in the living mouse was far better than what they would have expected from their experiments in a dish. They attributed this to the special cardiac environment and the presence of other cells in the heart that may have helped the fibroblasts convert to beating heart cells. However, another group of scientists attempted to replicate the findings of the 2010 Cell paper and found that their success rate was far lower than that of the Srivastava group. In the paper entitled “Inefficient Reprogramming of Fibroblasts into Cardiomyocytes Using Gata4, Mef2c, and Tbx5” published in the journal Circulation Research in 2012, Chen and colleagues found that very few fibroblasts could be converted into cardiomyocytes and that the electrical properties of the newly generated heart cells did not match up to those of adult heart cells. One of the key differences between this Circulation Research paper and the 2010 paper of the Srivastava group was that Chen and colleagues used fibroblasts from older mice, whereas the Srivastava group had used fibroblasts from newly born mice. Arguably, the use of older cells by Chen and colleagues might be a closer approximation to the cells one would use in patients. Most patients with heart attacks are older than 40 years and not newborns.

These studies were all performed on mouse fibroblasts being converted into heart cells, but they did not address the question whether human fibroblasts would behave the same way. A recent paper in the Proceedings of the National Academy of Sciences by Eric Olson’s laboratory (published online before print on March 4, 2013 by Nam and colleagues) has now attempted to answer this question. Their findings confirm that human fibroblasts can also be converted into beating heart cells, however the group of genes required to coax the fibroblasts into converting is slightly different and also requires the introduction of microRNAs – tiny RNA molecules that can also regulate the expression of a whole group of genes. Their paper also points out an important caveat.  The generated heart-like cells were not uniform and showed a broad range of function, with only some of the spontaneously contracting and with an electrical activity pattern that was not the same as in adult heart cells.

Where does this whole body of work leave us? One major finding seems to be fairly solid. Fibroblasts can be converted into beating heart cells. The efficiency of conversion and the quality of the generated heart cells – from mouse or human fibroblasts – still needs to be optimized. Even though the idea of cellular alchemy sounds fascinating, there are many additional obstacles that need to be overcome before such therapies could ever be tested in humans. The method to introduce these genes into the fibroblasts used viruses which permanently integrate into the DNA of the fibroblast and could cause genetic anomalies in the fibroblasts. It is unlikely that such viruses could be used in patients. The fact that the generated heart cells show heterogeneity in their electrical activity could become a major problem for patients because patches of newly generated heart cells in one portion of the heart might be beating at a different rate of rhythm than other patches. Such electrical dyssynchony can cause life threatening heart rhythm problems, which means that the electrical properties of the generated cells need to be carefully understood and standardized. We also know little about the long-term survival of these converted cells in the heart and whether the converted cells maintain their heart-cell-like activity for months or years. The idea of directly converting fibroblasts by introducing the genes into the heart instead of first obtaining the fibroblasts, then converting them in a dish and lastly implanting the converted cells back into the heart sounds very convenient. But this convenience comes at a price. It requires human gene therapy which has its own risks and it is very difficult to control the cell conversion process in an intact heart of a patient. On the other hand, if cells are converted in a dish, one can easily test and discard the suboptimal cells and only implant the most mature or functional heart cells.

This process of cellular alchemy is still in its infancy. It is one of the most exciting new areas in the field of regenerative medicine, because it shows how plastic cells are. Hopefully, as more and more labs begin to investigate the direct reprogramming of cells, we will be able to address the obstacles and challenges posed by this emerging field.

Image credit: Painting in 1771 by Joseph Wright of Derby - The Alchymist, In Search of the Philosopher’s Stone via Wikimedia Commons

Nam, Y., Song, K., Luo, X., Daniel, E., Lambeth, K., West, K., Hill, J., DiMaio, J., Baker, L., Bassel-Duby, R., & Olson, E. (2013). Reprogramming of human fibroblasts toward a cardiac fate Proceedings of the National Academy of Sciences, 110 (14), 5588-5593 DOI: 10.1073/pnas.1301019110

            Examples of the checklist questions are:

1. How was the sample size chosen to ensure adequate power to detect a pre-specified effect size? For animal studies, include a statement about sample size estimate even if no statistical methods were used.

5. For every figure, are statistical tests justified as appropriate? Do the data meet the assumptions of the tests (e.g., normal distribution)? Is there an estimate of variation within each group of data? Is the variance similar between the groups that are being statistically compared?

The authors are also reminded that they have to reveal complete statistical information in the figure legends and evidence that datasets have been submitted to public repositories for 1) Protein, DNA and RNA sequences, 2) Macromolecular structures, 3) Crystallographic data for small molecules and 4) Microarray data.

It is commendable that the Nature editors have recognized the importance of addressing the reproducibility issue in science, but I doubt that this checklist will make such a big difference. The cynical or perhaps overly honest answer to how many biologists determine sample size is not by a pre-specified sample size calculation. Instead, they might just go ahead and perform some arbitrary number of experiments with a sample size of n=5 or so, and then adjust the sample size by increasing it, if the initial results are not statistically significant until they achieve the equally arbitrary and near-mystical p-value thresholds of p<0.05 or p<0.01. This checklist will remind authors of the importance of keeping track of statistical and methodological details, and disclosing them in the manuscript. Such transparency in terms of methods and analyses is sorely needed. This will make it easier for other laboratories to attempt to replicate the published paper, but it is not clear how revealing these details will affect the chances that the results are indeed reproducible. Will the editors perhaps not review a manuscript if the checklist reveals that the authors only studied one strain of mice? Will sample sizes of n=5 not be acceptable even if the p-value is <0.01?

This brings us to another crucial point in the debate about reproducibility of scientific results. Prestigious journals such as Nature rarely review manuscripts that are deemed to be of limited significance or novelty to their readership. In fact, the vast majority of manuscripts submitted to high profile journals such as Nature of Science are rejected at the editorial level without ever undergoing a thorough peer review process. On the other hand, when editors get a personal call from high profile investigators, they may be more likely to send out a paper for review, because the publication of the paper could increase the often maligned "impact factor" of the journal.

Attempts to improve transparency and reliability of published research should not only target scientists, but also target the editorial and peer review process. Instead of the sending out a rather cryptic "Sorry, your paper is not interesting enough for us to review", shouldn't editors also complete a checklist that documents how they reached their decision? A checklist that addresses questions such as:

Was the acceptance/rejection of this manuscript based primarily on the scientific rigor or the number of expected citations per year?

How were the anonymous reviewers of this manuscript selected? How many of the chosen reviewers had been suggested by the authors? 

Did the authors directly interact with the editors to influence their decision whether or not to send a manuscript out for review?

Transparency is not a one-way mirror. Scientists need to become more transparent, but the editorial and review process should also be more transparent.

Image credit: Hall of Mirrors at Versailles (Image by Myrabella - Creative Commons License via Wikimedia)

1) Patients with a major heart attack (also referred to as ST-elevation Myocardial Infarction or STEMI) usually undergo an immediate angiogram of the heart to treat the blockage that is causing the heart attack by impeding the blood flow. This is the standard of care for heart attack patients in the developed world.

2) Patients enrolled in an experimental cell therapy trial are then brought back for a second procedure during which bone marrow is extracted with a needle under local anesthesia.

3) The research patients then undergo another angiogram of the heart using a catheter which allows for the infusion of bone marrow cells into the heart.

The hope is that the stem cells contained within the bone marrow are able to help regenerate the heart, either by turning into heart cells (cardiomyocytes), blood vessel cells (endothelial cells) or releasing factors that protect the heart and prevent the formation of a large scar. Unfortunately, there is very limited scientific evidence that bone marrow stem cells can actually turn into functional heart cells. The trials that have been conducted so far have yielded mixed results - some show that infusing the bone marrow cells indeed improves heart function, others show that patients who just receive the standard therapy with cell infusions do just as well. Most of the trials have been quite small - often studying only 10-50 patients.

The SWISS-AMI cell therapy trial, published online on April 17, 2013 in the world's leading cardiovascular research journal Circulation, addressed this question in a randomized, controlled trial, which enrolled 200 patients who had suffered a major heart attack. The published paper is entitled "Intracoronary Injection of Bone Marrow Derived Mononuclear Cells, Early or Late after Acute Myocardial Infarction: Effects on Global Left Ventricular Function" and was conducted in Switzerland.

The researchers assigned the patients to three groups: a) Standard heart attack treatment, b) Standard heart attack treatment and infusion of bone marrow cells 5-7 days after the heart attack or c) Standard heart attack treatment and infusion of bone marrow cells 3-4 weeks after the heart attack. They assessed heart function four months later using cardiac magnetic resonance imaging, one of the best tools available to determine heart function. The results were rather disappointing: Neither of the two cell treatment groups showed any improvement in their cardiac function.

This trial had some important limitations: Even though this study enrolled 200 patients and was thus larger than most other cell therapy trials for heart attack patients, it is still a rather small study when compared to other cardiovascular studies, which routinely enroll thousands of patients. Furthermore, this study only assessed heart function after four months and it is possible that if they had waited longer, they might have seen some benefit of the cell therapy. Despite these limitations, the trial will dampen the general enthusiasm for injecting bone marrow cells into heart attack patients.

Is this study a set-back for cardiac stem cell treatments? Not really. As the authors reveal in their data analysis, most of the cells contained in the bone marrow preparation that they used for the infusion were plain old white blood cells and NOT stem cells. Actually, only 1% of the infused cells were hematopoietic stem cells (stem cells that give rise to blood cells) and there was an undisclosed percentage of other stem cell types (such as mesenchymal stem cells) contained in the infused bone marrow extract. As I point out in the accompanying editorial "Bone Marrow Tinctures for Cardiovascular Disease: Lost in Translation", using such a mixture of poorly defined cells is ill-suited to promote cardiac regeneration or repair. Therefore, this important study is not a set-back for cardiac stem cell therapy, but a well-deserved setback for injections of undefined cells, most of which are not true stem cells!

Even if the majority of infused cells had been stem cells, there is no guarantee that merely infusing them into the heart would necessarily result in the formation of new heart tissue. Regenerating heart tissue from adult stem cells requires priming or directing stem cells towards becoming heart cells and ensuring that the cells can attach and integrate into the heart, not just infusing or injecting them into the heart.

It is commendable that the journal published this negative study, because too many treatments are being marketed as "stem cell therapies" without clarifying whether the injected cells are truly efficacious. Hopefully, the results of this trial will lead to more caution when rushing to perform "stem cell treatments" in patients without carefully defining the scientific characteristics and therapeutic potential of the cells that are being used.

Link to the original paper:  "Intracoronary Injection of Bone Marrow Derived Mononuclear Cells, Early or Late after Acute Myocardial Infarction: Effects on Global Left Ventricular Function"

Link to the editorial: "Bone Marrow Tinctures for Cardiovascular Disease: Lost in Translation"

Image credit: Surgeon extracting bone marrow from a patient (Public Domain image via Wikimedia)

Surder, D., Manka, R., Lo Cicero, V., Moccetti, T., Rufibach, K., Soncin, S., Turchetto, L., Radrizzani, M., Astori, G., Schwitter, J., Erne, P., Zuber, M., Auf der Maur, C., Jamshidi, P., Gaemperli, O., Windecker, S., Moschovitis, A., Wahl, A., Buhler, I., Wyss, C., Kozerke, S., Landmesser, U., Luscher, T., & Corti, R. (2013). Intracoronary Injection of Bone Marrow Derived Mononuclear Cells, Early or Late after Acute Myocardial Infarction: Effects on Global Left Ventricular Function Four months results of the SWISS-AMI trial Circulation DOI: 10.1161/CIRCULATIONAHA.112.001035

Rehman, J. (2013). Bone Marrow Tinctures for Cardiovascular Disease: Lost in Translation Circulation DOI: 10.1161/CIRCULATIONAHA.113.002775

Curating Creativity

 

"For every rational line or forthright statement there are leagues of senseless cacophony, verbal nonsense, and incoherency."

Jorge Luis Borges, "Library of Babel"

The British-Australian art curator Nick Waterlow was tragically murdered on November 9, 2009 in the Sydney suburb of Randwick. His untimely death shocked the Australian art community, not only because of the gruesome nature of his death – Waterlow was stabbed alongside his daughter by his mentally ill son – but also because his death represented a major blow to the burgeoning Australian art community. He was a highly regarded art curator, who had served as a director of the Sydney Biennale and international art exhibitions and was also an art ambassador who brought together artists and audiences from all over the world.

After his untimely death, his partner Juliet Darling discovered some notes that Waterlow had jotted down shortly before his untimely death to characterize what defines and motivates a good art curator and he gave them the eerily prescient title “A Curator’s Last Will and Testament”:

1. Passion

2. An eye of discernment

3. An empty vessel

4. An ability to be uncertain

5. Belief in the necessity of art and artists

6. A medium— bringing a passionate and informed understanding of works of art to an audience in ways that will stimulate, inspire, question

7. Making possible the altering of perception.

Waterlow’s notes help dismantle the cliché of stuffy old curators walking around in museums who ensure that their collections remain unblemished and instead portray the curator as a passionate person who is motivated by a desire to inspire artists and audiences alike.

 

You can read the complete essay here.

Biomedical research also produces a huge torrent of information and we are at risk of drowning in this vast ocean of scientific data. Our choices of what scientific papers we read, discuss or pass on to our colleagues are often quite arbitrary. My laboratory currently focuses on stem cell biology, especially in the context of cardiovascular differentiation and cardiovascular disease. When I searched the PubMed database for papers published in 2012 on "stem cells", the search identified 18,409 articles. A search with the keyword "cardiovascular" turned up 51,794 papers in 2012. Obviously no person can read and understand 70,000 research papers in a single year, therefore it is critical to "curate" and select the most relevant articles. Importantly, creative research ideas usually emerge when one is inspired by scientific ideas that lie outside of one's own area of research. A cardiovascular stem cell researcher in search for novel ideas should therefore not only read papers on cardiovascular research and stem cell biology, but also stay abreast of important developments in other areas, such as neurobiology, evolutionary biology, epigenetics or structural biology.

We all have developed our own personal ways of how we curate scientific content. We scan the table of contents of our favorite journals or receive email alerts from the journals, we may rely on scientific meetings and colleagues to inform us about new scientific developments or we browse science blogs - in most cases, our curatorial process is a combination of multiple approaches.

I hope that Scilogs readers will comment on their personal "algorithms" or methods of how they handle the science information glut in the comment section of this post.

How do you decide which biomedical research articles to read?

Do you primarily base your choices on PubMed keyword searches or table of contents / email notices from selected journals?

Do you regularly speak to colleagues or participate in journal clubs to identify important articles?

Is your choice of reading materials biased in favor of high-impact journals or high-impact researchers?

How do you choose articles you want to blog about, comment on or cite?

By sharing our experiences, we might be able to learn from each other, improve our curatorial skills and become better at managing the biomedical information deluge.

1. The Downside of Open-Access Publishing by Charlotte Haug

This article discusses potential problems associated with open access publishing but also conflates the issue of open access with the issue of inadequate peer review, as can be seen in this excerpt:

Of course, the terms “international,” “scientific,” “peer-reviewed,” “journal,” “article,” “editor,” and “publisher” do not have copyrighted or patented definitions and can have varied meanings, especially in the Internet age. Must an article be different from a submitted paper? Isn't everything published online automatically international? Is there anything wrong with a situation in which the editor and publisher are just one person who has set up a website where researchers can submit their papers and pay a fee to have them laid out in a professional way and made available to all interested parties? Isn't it a good thing that this vast number of new publishers and journals will make it possible to get all research — whatever its quality level — into the public domain? Perhaps. But describing a simple online-posting service as “an international, scientific, peer-reviewed journal” leads authors and readers to believe that they are submitting to or reading something they aren't.

One central flaw of this argument is that open access does not necessarily mean lack of peer review, as previously discussed.

2. Open but Not Free — Publishing in the 21st Century by Martin Frank

The article by Martin Frank tries to make the case that "open access" publishing itself costs quite a bit of money and that these funds could be better used for research purposes. He distinguishes between "gold open access", where published articles are immediately available upon publication to the general public without any fees for the readers and "green open access", which gives free access to the public after an initial period of pay-for-access. With green open access, the publisher generates some revenue during this initial period, whereas in "gold open access" publishing, the researchers usually pay a fee that covers the publication charges so that readers do not have to pay anything.

One section of the article especially caught my eye:

.....assuming that all articles had to be published with gold open access, Harvard Medical School would have to pay $13.5 million (at $1,350 per article) to publish the 10,000 articles authored by its faculty in 2010 — considerably more than the $3.75 million that was in its serials-acquisition budget that year. Research-intensive institutions will thus bear the burden of funding free access to the research literature, subsidizing access for less-research-intensive institutions, including pharmaceutical companies.

This calculation assumes that current pay-for-access journals do not charge researchers for the publication of their articles. I have previously addressed this issue, citing a specific example which shows that pay-for-access journals often charge the researchers several hundred dollars to publish an article. If researchers use color figures, the charges can run up to $2000 or $3000 per manuscript. These author fees are in addition to the fees that publishers of pay-for-access journals charge the readers. Martin Frank's calculation ignores the author fees that Harvard researchers might be currently paying to publish in pay-for-access journals.

He also mentions the pharmaceutical companies as potential beneficiaries but fails to include other important beneficiaries:

1) Members of the general public, whose taxes paid for most of the biomedical research conducted in the United States and who should thus have a right to access the results of this publicly funded research

2) Individuals in countries who cannot afford the fees to read papers published in pay-for-access journals.

3. Creative Commons and the Openness of Open Access by Michael Carroll

4. For the Sake of Inquiry and Knowledge — The Inevitability of Open Access by Ann Wolpert

In contrast, when it comes to the issue of global warming, there is a broad consensus in the scientific community. A 2010 study in the Proceedings of the National Academy of Sciences by Anderegg and colleagues reviewed published papers and statements made by climate researchers. The authors found that 97% to 98% of climate researchers were convinced by the scientific evidence for anthropogenic climate change, i.e. that humans are primarily responsible for global warming. When there is such a broad consensus among scientists and such overwhelming scientific data that supports anthropogenic climate change, one cannot really say "scientists are divided" merely because two or three scientists out of one hundred are not convinced.

Today, when I saw the headline "Scientists divided over device that 'remotely detects hepatitis C' " in the Guardian, I assumed that a major scientific study had been published describing a new way to diagnose Hepatitis C and that there was considerable disagreement among Hepatitis C experts as to the value of this new device. To my surprise, I found this description in the Guardian:

The device the doctor held in his hand was not a contraption you expect to find in a rural hospital near the banks of the Nile.

 For a start, it was adapted from a bomb detector used by the Egyptian army. Second, it looked like the antenna for a car radio. Third, and most bizarrely, it could – the doctor claimed – remotely detect the presence of liver disease in patients sitting several feet away, within seconds.

 The antenna was a prototype for a device called C-Fast. If its Egyptian developers are to be believed, C-Fast is a revolutionary means of using bomb detection technology to scan for hepatitis C – a strongly contested discovery that, if proven, would contradict received scientific understanding, and potentially change the way many diseases are diagnosed.

This "C-Fast device", co-developed by the Egyptian liver specialist Gamal Shiha, sounded like magic, and sure enough, even the Guardian referred to it as a "mechanical divining rod".

Witnessed in various contexts by the Guardian, the prototype operates like a mechanical divining rod – though there are digital versions. It appears to swing towards people who suffer from hepatitis C, remaining motionless in the presence of those who don't. Shiha claimed the movement of the rod was sparked by the presence of a specific electromagnetic frequency that emanates from a certain strain of hepatitis C.

After I read the remainder of the article, it turned out there are no published scientific studies to confirm that this rod, antenna or wand can detect hepatitis viruses at a distance.  The article says it "has been successfully trialled in 1,600 cases across three countries, without ever returning a false negative result", but this data has not been published in a peer-reviewed journal. As a scientist and a physician, I am of course very skeptical. The physicians using this device claim it has 100% sensitivity without presenting the data in a peer-reviewed forum. But what is even more surprising is the suggestion that electromagnetic signals travel from the virus in the body of a patient to this remote device, without any scientific evidence to back this up.

The Guardian then also quotes a University College London expert:

"If the application can be expanded, it is actually a revolution in medicine," said Pinzani, head of UCL's liver institute. "It means that you can detect any problem you want."

 By way of example, Pinzani said the device could conceivably be used to instantaneously detect certain kinds of cancer symptoms: "You could go into a clinic, and a GP could find out if you had a tumour marker."

This expert is already fantasizing about cancer diagnostics with this divining rod even though there is no credible published scientific data. The Guardian article also mentions that well-known scientific journals have rejected articles about this new device and that the "scientific basis has been strongly questioned by other scientists", but the Guardian is compromising its journalistic integrity by presenting this as a legitimate scientific debate and claiming that "scientists are divided" in the title of the article. How can scientists be divided if the data has not been made public and if it has not undergone peer review? For now, this claim of a diagnostic divining rod is pure sensationalism and not an actual scientific controversy. Such sensationalism will attract many readers, but it should not be an excuse for shoddy journalism.

Image Credit: Public domain image of Otto Edler von Graeve in 1913 with a divining rod via Wikimedia Commons

UPDATE: The comment thread of the Guardian article indicates that Pinzani feels misrepresented by the article and cites a letter that Pinzani has purportedly written in response to the article. I am not able to verify whether this letter was indeed written by him and how exactly Pinzani was misrepresented by the Guardian.

UPDATE February 26, 2012: The Guardian has now changed the headline to Scientists sceptical about device that 'remotely detects hepatitis C'. I think this headline is much better than the previous one which suggested that "scientists were divided". I still think that newspapers and magazines sometimes unnecessarily portray pseudo-scientific viewpoints as legitimate, equal partners in a scientific debate. This type of even-handedness only makes sense if certain viewpoints are backed up by rigorous scientific studies.

Soon after the discovery of DNA, the primary function ascribed to DNA was its role as a template from which messenger RNA could be transcribed and then translated into functional proteins. Using this definition of "function", only 1-2% of the human DNA would be functional because they actually encoded for proteins. The term "junk DNA" was coined to describe the 98-99% of non-coding DNA which appeared to primarily represent genetic remnants of our evolutionary past without any specific function in our present day cells.

However, in the past decades, scientists have uncovered more and more functions for the non-coding DNA segments that were previously thought to be merely "junk". Non-coding DNA can, for example, act as a binding site for regulatory proteins and exert an influence on protein-coding DNA. There has also been an increasing awareness of the presence of various types of non-coding RNA molecules, i.e. RNA molecules which are transcribed from the DNA but not subsequently translated into proteins. Some of these non-coding RNAs have known regulatory functions, others may not have any or their functions have not yet been established.

Despite these discoveries, most scientists were in agreement that only a small fraction of DNA was "functional", even when all the non-coding pieces of DNA with known functions were included. The bulk of our genome was still thought to be non-functional. The term "junk DNA" was used less frequently by scientists, because it was becoming apparent that we were probably going to discover even more functional elements in the non-coding DNA.

In September 2012, everyone was talking about "junk DNA" again, because the ENCODE scientists claimed their data showed that 80% of the human genome was "functional". Most scientists had expected that the ENCODE project would uncover some new functions for non-coding DNA, but the 80% figure was way out of proportion to what everyone had expected. The problem was that the ENCODE project used a very low bar for "function". Binding to the DNA or any kind of chemical DNA modification was already seen as a sign of "function", without necessarily proving that these pieces of DNA had any significant impact on the function of a cell.

The media hype with the "death of junk DNA" headlines and the lack of discussion about what constitutes function were appropriately criticized by many scientists, but the recent paper by Dan Graur and colleagues "On the immortality of television sets: “function” in the human genome according to the evolution-free gospel of ENCODE" has grabbed everyone's attention. Not necessarily because of the fact that it criticizes the claims made by the ENCODE scientists, but because of the sarcastic tone it uses to ridicule ENCODE.

There have been so many other blog posts and articles that either praise or criticize the Graur paper, so I decided to list some of them here:

1. PZ Myers writes "ENCODE gets a public reaming" and seems to generally agree with Graur and colleagues.

2. Ashutosh Jogalekar says Graur's paper is a "devastating takedown of ENCODE in which they pick apart ENCODE’s claims with the tenacity and aplomb of a vulture picking apart a wildebeest carcass."

3. Ryan Gregory highlights some of the "zingers" in the Graur paper

Other scientists, on the other hand, agree with some of the conclusions of the Graur paper and its criticism of how the ENCODE data was presented, but disagree with the sarcastic tone:

1. OpenHelix reminds us that this kind of "spanking" should not distract from all the valuable data that ENCODE has generated.

2. Mick Watson shows how Graur and colleagues could have presented their key critiques in a very non-confrontational manner and foster a constructive debate.

3. Josh Witten points out the irony of Graur accusing ENCODE of seeking hype, even though Graur and his colleagues seem to use sarcasm and ridicule to also increase the visibility of their work. I think Josh's blog post is an excellent analysis of the problems with ENCODE and the problems associated with Graur's tone.

On Twitter, I engaged in a debate with Benoit Bruneau, my fellow Scilogs blogger Malcolm Campbell and Jonathan Eisen and I thought it would be helpful to share the Storify version here. There was a general consensus that even though some of the points mentioned by Graur and colleagues are indeed correct, their sarcastic tone was uncalled for. Scientists can be critical of each other, but can and should do so in a respectful and professional manner, without necessarily resorting to insults or mockery.

Graur D, Zheng Y, Price N, Azevedo RB, Zufall RA, & Elhaik E (2013). On the immortality of television sets: "function" in the human genome according to the evolution-free gospel of ENCODE. Genome biology and evolution PMID: 23431001