A Crossroads in Biomedicine?
Scientific knowledge is supposed to be built piecewise—study by study, experiment by experiment—so that each new piece of knowledge rests on a firm foundation of preceding work. Discovering that influential pieces of research cannot, in fact, be replicated by other researchers in other labs is akin to discovering a foundation riddled with holes, but that’s exactly what has happened in the last few decades, in fields from chemistry to psychology. This “reproducibility crisis” has prompted soul-searching throughout the sciences, as researchers reconsider what they know and how they know it.
How did this happen? Nicole C. Nelson, an assistant professor of science and technology studies at the University of Wisconsin-Madison and a visiting fellow at Harvard’s Radcliffe Institute for Advanced Study, is well situated to find out. An ethnographer and historian of science, Nelson studies the assumptions that scientists make about the world and how that affects their work. She is spending her fellowship year researching the origins of the replication crisis in pre-clinical research: the experiments on animals and cell cultures that inspire the development of new pharmaceuticals.
On Wednesday night at Radcliffe, Nelson discussed the origins of the reproducibility crisis in biomedicine and proposed that any solution would require not only changing how scientists talk about uncertainty but also confronting the power dynamics in academic labs that can exacerbate reproducibility concerns. The talk was part of the institute’s Fellows’ Presentation Series, which brings the work of its 50 scholars—from across the arts, humanities, sciences, and social sciences—to the public.
Despite the sense in the last decade or so that the reproducibility problem is an emerging crisis, Nelson sees it as a continuation of reform efforts that began in the field of clinical medicine in the 1990s. To prevent unscrupulous drug companies or journals interested only in the flashiest results from biasing the scientific record, nearly all clinical trials since the early 2000s have been registered before they even begin. This ensures that their existence is made public, whether their results show the treatment to be beneficial, ineffective, or dangerous.
At the same time, clinicians have adopted a standardized ranking system to express the strength of evidence supporting different therapies. Grades show which conditions an intervention benefits, what kind of trials the evidence comes from, and in which patient populations the evidence is strongest, making it clearer to clinicians how confident they can be that a given therapy will work for their specific patient. Using such a system effectively, said Nelson, means “thinking about how to talk about scientific data in ways that don’t rely on the absolutist notion of truth, but instead emphasize those shades of gray.”
But these strategies have not yet permeated research done using cells and animals, she continued. Therefore, when research teams at both Amgen and Bayer tried to reproduce dozens of different research results in the early 2010s, and found that only a small percent could actually be duplicated, the field was ripe for a reckoning.
Though the need for reform is clear, Nelson conceded that the path forward is trickier in pre-clinical research than in clinical trials. “The pharmaceutical industry makes a really good villain,” she explained: the profit motive creates a simple narrative and allows academics and others outside the industry to push for relatively simple changes. In comparison, “it’s undoubtedly more difficult for a community to look critically at its own practices.” In academic labs, she said, the question of what drives a bias toward positive results is more complicated. “Is it because of pressure to publish from funders, from your tenure committee, from journals that want to publish novel results, or is it internal?”
Whatever the cause, Nelson suggested that pre-clinical scientists could mitigate the effect by borrowing language from clinical research to say explicitly, first, how confident they are in their results and, second, under which conditions these results ought to be replicable. Systematic tracking of which cell cultures or lab animals a treatment has been tested on—and whether the therapy ameliorates the actual condition or a close (or not-so-close) proxy for it—would allow scientists to deal with uncertainty more comfortably and consistently, she said.
Although scientists within the field often seek external solutions, such as more funding to replicate studies or better statistical education for trainees, Nelson said the field will have to do some self-reflection. For instance: it’s all too easy for professors to assume that students’ failures to replicate past work is an error driven by inexperience, and therefore push them to tweak their methods and analysis until they can coax out “successful” results.
This pressure, Nelson pointed out, will fall more heavily on underrepresented minorities whose competence is less likely to receive the benefit of the doubt. “Taking this problem seriously requires talking about power differentials within the lab, and requires confronting the idea that science is not separate from society with all of its issues and social biases,” she concluded. “It reminds us that science is done by humans.”