A Four Part Series on Open Notebook Science (Part 1)

1 January 2014 by Shannon Bohle, posted in Uncategorized

As a librarian and archivist, I have worked with a variety of materials that record scientific advancements, such as books, journal articles, letters of correspondence, photos, x-rays, and laboratory notebooks. In a four-part article series, I would like to emphasize the importance of preserving laboratory notebooks. In particular, I will firstly examine the role laboratory notebooks play assisting the reproducibility of scientific results, secondly, discuss the changing value of laboratory notebooks in the newly implemented "first to file" patenting process in the U. S., thirdly, debate models for creating an ethos of openness for notebook contents, and finally, consider the United States Patent and Trademark Office (USPTO) as a possible central depository for a large number of laboratory notebooks that could accompany U. S. patent filings.

Why Reproducibility, Not Peer Review, is the Gold Standard in Science

The reticence and fear of scientists to admit to or confess to the discovery of error (whether clear or arguable) is understandably human. But even Nobel laureates and some of the other most brilliant minds in science have made huge blunders. Astrophysicist and science popularizer Mario Livio writes about key mistakes made by five of the most famous scientists in modern history: Darwin, Einstein, Hoyle, Kelvin, and Pauling in his bestselling book, Brilliant Blunders (New York: Simon & Schuster, 2013). He writes, “mistakes in any discipline that is based on creative thinking and innovation are not only inevitable, they are an ESSENTIAL part of progress. If not for blunders, we would be traveling for too long down too many blind alleys. Breakthroughs require the willingness to embrace risks and to accept errors as potential portals of discovery.”

Mars Climate Orbiter. Image Credit: NASA.

Mars Climate Orbiter. Image Credit: NASA.

Modern magnificent miscalculations have also grabbed headlines. For example, in 1999, NASA’s $193.1 million USD Mars Climate Orbiter spacecraft was destroyed because a team of rocket scientists failed to convert measurements from English (pounds) to metric (newtons): “‘Our inability to recognize and correct this simple error has had major implications,’ said JPL director Edward Stone.”

The problem is not that scientists make inadvertent mistakes, but that those mistakes are not caught before it is too late, and “too late” means serious financial investments or exposure to health risks. Earlier this year, open access publishing reviewers were put to the test. “John Bohannon, a science journalist at Harvard University, saw various versions of a bogus scientific paper being submitted to 304 open access journals worldwide over a period of 10 months … [and] 45% of Directory of Open Access Journals (DOAJ) publishers that completed the review process, accepted the paper.” This brings attention to a serious criticism to modern scientific methods for establishing facts and findings. Reproducibility, not peer review, is the gold standard for determining whether science is “science” or “bunk.”

As detailed later in this article, there have also been serious mistakes made in established and respected subscription peer reviewed journals, not just those published through an open access model. Greater transparency, through the provision of underlying data and methods, is the solution that is being sought to help prevent these types of costly errors. Funded projects are now being designed to test the reproducibility of findings in landmark papers.

Laboratory Notebooks and the Reproducibility of Scientific Results

This first article in the series covers the role of laboratory notebooks on reproducibility and replication in the sciences. Laboratory notebooks help document the day-to-day thoughts and activities of scientists and, along with data sets, could complement results published in journals and patent filings by demonstrating experimental procedures in detail. Laboratory notebooks trace each step leading to discovery thereby enabling the scrutiny of scientific method for validity, accuracy, and reproducibility.

A variety of books, academic studies, and examples published in the media are sending the message that published journal articles alone are just not enough any more. Take, for example, this article published in The Economist, which pointed out serious flaws in today's non-reproducible science journal articles and the role that virtual laboratory notebooks could play in monitoring experimental studies:

Some government funding agencies, including America’s National Institutes of Health, which dish out $30 billion on research each year, are working out how best to encourage replication … Ideally, research protocols should be registered in advance and monitored in virtual notebooks … A rule of thumb among biotechnology venture-capitalists is that half of published research cannot be replicated. Even that may be optimistic. Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 “landmark” studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three-quarters of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients took part in clinical trials based on research that was later retracted because of mistakes or improprieties.

When Data is Restricted, Notebooks Provide a Viable Alternative

Despite the need for original data to reproduce the results of scientific and medical studies, there are times that laboratory notebooks might be valuable when other forms of data should be withheld. Indeed, there are times when the public, science, and the economy benefit by the withholding of information. Privacy is one reason to withhold information. A second is security. Data security is particularly problematic when considering the problem of the Mosaic Effect.

The mosaic effect occurs when the information in an individual dataset, in isolation, may not pose a risk of identifying an individual (or threatening some other important interest such as security), but when combined with other available information, could pose such risk. Before disclosing potential [Personally Identifiable Information (PII)] or other potentially sensitive information, agencies must consider other publicly available data – in any medium and from any source – to determine whether some combination of existing data and the data intended to be publicly released could allow for the identification of an individual or pose another security concern.

In these situations, laboratory notebooks can be useful for reproducibility because they contain valuable information regarding specific methodological approaches, analytical interpretations of data, and so on, without actually needing to reveal data sets themselves. Notebooks are also more easily redacted than data sets.

The health industry, in particular, faces a four-fold problem relating to the public dissemination of research study results, let alone the underlying data. Firstly, securing data usage rights from patients for use in studies, secondly, anonymizing and protecting those data after they are generated, thirdly, choosing between the right model (open, such as with a biotechnology patent, or closed, like with software trade secrets), and, finally, fearing litigation should disseminated results be challenged in which, during the legal process, the need to produce those data sets and laboratory notebooks as evidence of research claims arises.

A  scientist examines a slide under a microscope. Image credit: National Cancer Institute.

A scientist examines a slide under a microscope.
Image credit: National Cancer Institute.

In a special issue on “Communication in Science: Pressures and Predators,” published recently in Science, case examples arose wherein the problem was raised that publicly “disseminating certain scientific information could pose a threat to safety and security” such as “the recent debate over whether to publish influenza gain-of-function studies,” because they contain biological and chemical details that might be used to create weapons. (R. Stone, B. Jasny, Science 342, 57 (2013); D. Malakoff, Science 342, 70-71 (2013)). Another example from that same journal issue is related to protecting personal patient data. The biotechnology firm, Amgen, acquired the Icelandic company deCODE “after Iceland’s Supreme Court said that deCODE could not use the health records without consent” and it filed for bankruptcy (J. Kaiser, Science 340, 1388-1389 (2013)). If, as a policy, patients begin using the legal system to retroactively prevent the use of collected data (like they did with deCode) or to increase patient data rights, it could prove problematic to any company that chooses to use those data for published studies should those data be retroactively restricted. Without these underlying data, study results may not be able to be duplicated independently and could invalidate the published findings.

In the meantime, the U. S. tightened its restrictions to patient health data use and re-sale. The 2013 Health Insurance Portability and Accountability Act (HIPAA) updates made an effort to strengthen Protected Health Information (PHI) through improved “physical, administrative, and technical safeguards” via risk assessments as well as extending those protections to “business associates.” As certain types of data become less available for clinical studies or become over-regulated through HIPAA amendments and other added legal privacy protections as an era of "big data" evolves, laboratory notebooks begin to seem like an appealing alternative. The editors of the special issue of Science balanced both sides of the story with its selection of articles, and provided an overwhelming sense that sometimes it is necessary for multiple stakeholder groups “to balance an ethos of openness with demands for secrecy" because some scientific studies have "findings so sensitive that scientists can spend countless hours fretting over when, where, and how to publish them – or whether to publish them at all” (D. Malakoff, Science 342, 70 (2013)).

Economic considerations are another reason to withhold information. Deciding between a trade secret and a patent is a common conundrum for inventors and industry. “In principle, the U. S. patent system is intended to encourage the free flow of new knowledge so that society can benefit,” but “in practice … the system can work against innovators” by “lock[ing] away data,” as exemplified in the recent Supreme Court case Association for Molecular Biology v. Myriad (2013) (E. Marshall, Science 342, 72 (2013)). Such that “scientists [working] in industry, too, are struggling to define the limits of openness when communicating proprietary research, and whether some kinds of patents may actually squelch innovation” (R. Stone, B. Jasny, Science 342, 57 (2013)).

When Data is Partial or Contradictory, Laboratory Notebooks Fill in Gaps

Under the new open data mandates, best practices for published papers should include “cleaned” data sets, as well as the methods and software used to obtain the data. These published data sets are processed and edited rather than consisting of raw data, and sometimes they rely on assumptions and detailed methods that are not always made clear.

As of right now there are no clear incentives to actually reproduce research, and in some cases the lack of equipment, time, and funding do not support the centuries-old concept of reproduction of results but rather rely on waiting for new research to emerge that contradicts the old. Yet the visibility of problems in reproducibility is huge once papers are put under the microscope, so to speak. A study published in Nature Genetics examined 18 papers and found that only two could be reproduced “in principle,” six could be reproduced “partially or with some discrepancies,” and ten “could not be reproduced; “the main reason for failure to reproduce was data unavailability, and discrepancies were mostly due to incomplete data annotation or specification of data processing and analysis” (J. P. Ioannidis et al., Nat Genet. 41, 149-155 (2009)). Major data errors were also found to be present in a 2006 published paper by researchers Potti et al. in Nature Medicine who claimed to have found a new method for predicting the efficacy of ovarian cancer targeted drug therapies (A. Potti et al., Nat. Med. 12, 1294-1300 (2006)). But when scrutinized in a study by Baggerly and Coombs, data errors (or falsifications) were made by the experimenters such that their method would not work and the paper had to be retracted in 2011 (K. A. Baggerly, K. R. Coombes, Ann. Appl. Stat. 3.4, 1309-1334 (2009); A. Potti et al., Nat. Med. 12, 1294-1300 (2006)). The consequences to the public are that “poor documentation and irreproducibility can shift from an inconvenience to an active danger when it obscures not just methods but errors. This can lead to scenarios where well-meaning investigators argue in good faith for treating patients with apparently promising drugs that are in fact ineffective or even contraindicated” (K. A. Baggerly, K. R. Coombes, Ann. Appl. Stat. 3.4, 1309-1334 (2009)). Geneticist Elisabeth Iorns says, “The lack of reproducibility in cancer studies is a major obstacle in the development of viable therapies to cure cancer,” and that “the funding [of the Reproducibility Initiative] will be instrumental in not only verifying cancer studies, but also helping to institutionalize scientific replication.”

As published data sets become more common, there are important benefits to having access to the associated laboratory notebooks. When data are only partial or contradictory, open laboratory notebooks could fill in gaps without having to contact the researcher directly. Even if the researcher is contacted, it is unlikely they will have perfect recall and it is possible they might not have preserved the notebook needed to help fill in their memory gaps. Even in the absence of "cleaned" data sets or lack of the full, raw data, the advantage of having access to the researcher's notebooks is that they usually contain at least some of the raw data (and possibly all raw data linked to them from instruments) as well as intermediate values. This allows examiners to determine the quality of execution of the methodology or to apply new processing techniques at a later date, potentially getting different, better results.

As Peer Review Evolves to Accommodate More Data,
Will There Be Room for Laboratory Notebooks?

From a journal's perspective, under expanded open data policies, it remains uncertain how peer review can be maintained in a manageable way and whether reproducibility requirements for evaluating supporting data sets (with or without their associated laboratory notebooks) might even be feasible given the sheer magnitude of papers selected for publication. Implementing an added layer to the peer review process by checking reproducibility presents formidable problems and expense, and might stress an already overburdened peer review system to the point of breaking.

Right now, peer-reviewers often lack the data to evaluate submissions, and simply operate on their best professional judgment. Or, in cases of single-blind or non-blind submissions, the reputations of the submitting scientist or their institution may also be taken into consideration. On the other hand, in too many cases, when journals are provided "cleaned" data sets, the software needed for reproduction by peer reviewers is not named or is not open source so that it is not possible to replicate that data or simulation free of charge to arrive at the same results (V. Stodden, P. Guo, Z. Ma, PLOS ONE 8.6, e67111 (2013)). There is also the added problem that raw data from specific measuring devices often requires knowledge of particular instruments for the data to be useful that a peer reviewer might not possess. In all of these cases, having a copy of the related laboratory notebooks would be useful to the peer reviewer. However, scientists are reluctant to take on the additional work needed to prepare a notebook for publication. They may have additional concerns about how preliminary experiments and wrong turns prior to arriving at a solution might affect the reviewer's opinion of their credibility. Scientists might also be reluctant to offer their notebooks to peer reviewers in the event that some of these unrelated materials might not be ready for publication or patenting and information could be inadvertently revealed too soon. Nevertheless, it is possible that these concerns would be offset by the benefits. Open notebooks science would encourage scientists and their students to be more meticulous and employ best practices when keeping their notebooks. For example, the importance of naming the instruments used and descriptions of those instruments for aiding in reproducibility. In a well-organized notebook or series of notebooks, for example, reviewers would not have difficulty filtering through them solely for the experiment at hand, given that a variety of unrelated experiments are often spread throughout the same notebook. In the case of electronic notebooks in a paperless digital lab (J. Giles, Nature 481, 430-431 (2012)), adding explanatory metadata could be of exceptional importance for aiding reproducibility.

In the future, under ideal conditions, a publishers could use a hypertext link to connect a published paper with its underlying data, laboratory notebooks, images, and related patents. Automated scientific data mining and analytical techniques may prove to re-energize the process of reproducibility in the long-term. For the short-term, projects like the Open Science Initiative, which received $1.3M "to independently validate 50 landmark cancer biology studies," and encouraging graduate students to undertake these types of projects might help offset the burden placed on journals and peer reviewers.

Studies chosen to participate in the Reproducibility Initiative and will undergo validation studies. Reproducibile studies will "have the option to publish the replicated results as an independent publication in the PLOS ONE Reproducibility Collection, and can share the data via the Figshare Reproducibility Collection repository."

In a more open working climate, one that expects and knows how to deal with error, open notebook science may actually inspire greater innovation. This is because “a raw idea, can really be much more inspiring than the cleaned up peer-reviewed idea presented a year later” (N. Stafford, Nature 467, S19-S21 (2010)). Additionally, greater transparency of underlying data may also speed the rate of scientific advancement by supporting greater risk taking amid a risk-averse scientific culture. If everything is transparent, and the data are clearly supportive and can be reproduced by third parties, bolder (as opposed to incremental) claims can more confidently be asserted. The result is a new vision for the collaborative future of science: “Rather than spending time collecting their own data, scientists will organize themselves around shared data sets” (D. Butler, Nature 402 (6761 Suppl), C67-70 (1999)). With the availability of laboratory notebooks, they will not only have the data sets but be able to competently re-use them.

The Perceived Value of Laboratory Notebooks is Underrated

Will all this work be worth the effort? There are certainly potential drawbacks to implementing both open data and open notebook science policies. In the long-run, the value of open notebooks will need to balance those drawbacks if the concept will flourish. Under existing conditions, particularly those in the peer review system mentioned earlier as well as mental "blocks" within the minds of scientists, they do not.

When it comes to sharing laboratory notebooks openly, a famous phrase from the movie Cool Hand Luke comes to mind about the stubborn prisoner who does not want to do what he is told. Despite being told the benefits of data sharing, many scientists, imprisoned by a variety of fears, hoard their data and avoid sharing them at all if possible — with the public, their university, or even other teams in the same department — especially when it comes to their laboratory notebooks. Unfortunately, years later, many scientists destroy most of their notebooks and they are lost to history. Recently a new study published in the journal Current Biology found that "odds of obtaining an original data set...[falls] by 17 percent each year" and that "by 20 years post-publication" 80% of that data obtained from publicly-funded research is inaccessible." Their solution was to put out a similar call for mandates, asking "journals to require that authors share their data on a public archive before a paper can be published."

Generally, scientists who are not Nobel laureates may recognize the primary value of their own materials to their own research and to their employers, but remain somewhat oblivious to the secondary values of the materials they produce. There is an underlying misconception that unless their work has produced some famous result that no one would want to read the notebooks of an “average” scientist. One unnamed, experienced scientist even pointed out to me that if case studies are needed for educational purposes, the notebooks of Nobel laureates or other eminent individuals should be used. This is despite the fact that Nobel laureates have already responded with a decisive answer to this. In 2007, two scientists, Nobel laureates in Physiology or Medicine, Sydney Brenner and Rich Roberts, wrote a correspondence to Nature with a plea to scientists to “Save your notes, drafts, and printouts,” arguing, “science is one of the greatest cultural achievements of humankind. And yet...there is little systematic preservation of the workings of scientists” (S. Brenner, R. Roberts, Nature 446, 725 (2007)). Over the years, scientists like this one have made several comments to me personally when we casually discussed the potential for widespread publishing of laboratory notebooks. The overwhelming feeling I got from these scientists was their belief that the more important the discovery, the more interested people will be to read about the details. However, that simply is not the case. The fact is, from my informal interviews and discussions with them, scientists (and science students) do not seem to be aware of the non-scientist stakeholders who value their notebooks and cannot envision the potential re-use of their notebook materials for other constructive purposes that they themselves would not have thought up. In short, scientists need to discern that they do not have to be Nobel laureates for people to be interested in their notebooks.

Perhaps because scientists think of big problems they subconsciously overvalue their own insignificance. Image credit: NASA/NOAA/GSFC/Suomi NPP/VIIRS/Norman Kuring.

Perhaps because scientists think of big problems they subconsciously overvalue their own individual insignificance.
Image credit: NASA/NOAA/GSFC/Suomi NPP/VIIRS/Norman Kuring.

Indeed, notebooks and data sets might be utilized to create data tools and visualizations, and have nothing at all to do with the “significance” of a discovery from which the notebook was derived, but merely the patterns in data. These patterns may lead to a new discovery. Others interested in the notebooks might include those engaged in scientific and budgetary oversight, investigators of scientific fraud, other scientific researchers, historians of science, secondary school educators, museums wishing to link sample data output to instruments in their collections, computer scientists and innovators working in the data sciences (who want linked data, metadata extraction to apply context for data, data re-use, data modeling and data visualization), and funders who want greater budgetary oversight and transparency, as well as the public. Perhaps what is most interesting are the potential uses not yet imagined by making laboratory notebook resources openly available.

Despite what scientists might think, the public is interested in reading about their work and becoming more involved in science, even those nitty-gritty details. The popularity of citizen science projects has been one example of this. I cannot think of more nitty-gritty things than labeling thousands of solar flares from images, for example, (yes, I did this myself), or classifing 30 years of tropical cyclone data. Even though lab notebooks might be difficult for the public to understand, science communicators might be able to act as an intermediary. It is important to keep in mind that public participation in science is not limited to repetitive, mundane activities that require little thought, but also emcompasses the ability to understand and contribute to "the big picture." That is, members of the public can offer ideas that might help shape the future direction of scientific policy and intitatives. A recent post on The Guardian's web site pointed out that, "Non-scientists [can] influence the course of scientific research ... Science communication should be more than the dissemination of results to the public; it should also flow in the other direction, with members of the public able to communicate their priorities to scientists and those who fund them."

Even when scientists feel bombarded by pseudoscience, conspiracy theories, and a low level of scientific understanding in the public, they should embrace those opportunities. Asking challenging questions, and being open to seeking out the answers and presented evidence, is the first step towards scientific thinking. Experts disagree, and mistakes are made on both sides, so noting these appropriately and not presenting false/inaccurate findings, whether positive or negative, is essential. Performing an “under the hood” diagnostics with the workings of “the publishing machine,” through the process of reproducibility, can help tweak internal problems and advance scientific progress. Transparency and reproducibility can also help calm public concerns and provide assurances that the public is not being mislead by government or corporations who fund the studies.

One group of non-scientists who possess a long history recognizing the secondary values of laboratory notebooks is patent litigators. In a court of law, laboratory notebooks have been used in patent litigation regarding the inventorship of discoveries and claims of those discoveries. However, a recent change in US patent law and policy making is now affecting U. S. scientists and patent filers, and this may affect the value of laboratory notebooks both in the courtroom and in the laboratory. Part two of this series will look closely at the changing legal value of laboratory notebooks in light of the American Invents Act.

Leave a Reply

two + 6 =