Sloppy Science Happens More Than You Think
The media loves to tout scientific breakthroughs, and few are as toutable – and in turn, have been as touted – as CRISPR. This method of targeted DNA excision was discovered in bacteria, which use it as an adaptive immune system to combat reinfection with a previously encountered virus.
Shouldn't the editors at a Nature journal know better than to have published an incorrect paper in the first place?
It is cool on so many levels: not only is the basic function fascinating, reminding us that we still have more to discover about even simple organisms that we thought we knew so well, but the ability it grants us to remove and replace any DNA of interest has almost limitless applications in both the lab and the clinic. As if that didn't make it sexy enough, add in a bicoastal, male-female, very public and relatively ugly patent battle, and the CRISPR story is irresistible.
And then last summer, a bombshell dropped. The prestigious journal Nature Methods published a paper in which the authors claimed that CRISPR could cause many unintended mutations, rendering it unfit for clinical use. Havoc duly ensued; stocks in CRISPR-based companies plummeted. Thankfully, the authors of the offending paper were responsible, good scientists; they reassessed, then recanted. Their attention- and headline- grabbing results were wrong, and they admitted as much, leading Nature Methods to formally retract the paper this spring.
How did this happen? Shouldn't the editors at a Nature journal know better than to have published this in the first place?
Alas, high-profile scientific journals publish misleading and downright false results fairly regularly. Some errors are unavoidable – that's how the scientific method works. Hypotheses and conclusions will invariably be overturned as new data becomes available and new technologies are developed that allow for deeper and deeper studies. That's supposed to happen. But that's not what we're talking about here. Nor are we talking about obvious offenses like outright plagiarism. We're talking about mistakes that are avoidable, and that still have serious ramifications.
The cultures of both industry and academia promote research that is poorly designed and even more poorly analyzed.
Two parties are responsible for a scientific publication, and thus two parties bear the blame when things go awry: the scientists who perform and submit the work, and the journals who publish it. Unfortunately, both are incentivized for speedy and flashy publications, and not necessarily for correct publications. It is hardly a surprise, then, that we end up with papers that are speedy and flashy – and not necessarily correct.
"Scientists don't lie and submit falsified data," said Andy Koff, a professor of Molecular Biology at Sloan Kettering Institute, the basic research arm of Memorial Sloan Kettering Cancer Center. Richard Harris, who wrote the book on scientific misconduct running the gamut from unconscious bias and ignorance to more malicious fraudulence, largely concurs (full disclosure: I reviewed the book here). "Scientists want to do good science and want to be recognized as such," he said. But even so, the cultures of both industry and academia promote research that is poorly designed and even more poorly analyzed. In Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Millions, Harris describes how scientists must constantly publish in order to maintain their reputations and positions, to get grants and tenure and students. "They are disincentivized from doing that last extra experiment to prove their results," he said; it could prove too risky if it could cost them a publication.
Ivan Oransky and Adam Marcus founded Retraction Watch, a blog that tracks the retraction of scientific papers, in 2010. Oransky pointed out that blinded peer review – the pride and joy of the scientific publishing enterprise – is a large part of the problem. "Pre-publication peer review is still important, but we can't treat it like the only check on the system. Papers are being reviewed by non-experts, and reviewers are asked to review papers only tangentially related to their field. Moreover, most peer reviewers don't look at the underlying or raw data, even when it is available. How then can they tell if the analysis is flawed or the data is accurate?" he wondered.
Mistaken publications also erode the public's opinion of legitimate science, which is problematic since that opinion isn't especially high to begin with.
Koff agreed that anonymous peer review is valuable, but severely flawed. "Blinded review forces a collective view of importance," he said. "If an article disagrees with the reviewer's worldview, the article gets rejected or forced to adhere to that worldview – even if that means pushing the data someplace it shouldn't necessarily go." We have lost the scientific principle behind review, he thinks, which was to critically analyze a paper. But instead of challenging fundamental assumptions within a paper, reviewers now tend to just ask for more and more supplementary data. And don't get him started on editors. "Editors are supposed to arbitrate between reviewers and writers and they have completely abdicated this responsibility, at every journal. They do not judge, and that's a real failing."
Harris laments the wasted time, effort, and resources that result when erroneous ideas take hold in a field, not to mention lives lost when drug discovery is predicated on basic science findings that end up being wrong. "When no one takes the time, care, and money to reproduce things, science isn't stopping – but it is slowing down," he noted. Mistaken publications also erode the public's opinion of legitimate science, which is problematic since that opinion isn't especially high to begin with.
Scientists and publishers don't only cause the problem, though – they may also provide the solution. Both camps are increasingly recognizing and dealing with the crisis. The self-proclaimed "data thugs" Nick Brown and James Heathers use pretty basic arithmetic to reveal statistical errors in papers. The microbiologist Elisabeth Bik scans the scientific literature for problematic images "in her free time." The psychologist Brian Nosek founded the Center for Open Science, a non-profit organization dedicated to promoting openness, integrity, and reproducibility in scientific research. The Nature family of journals – yes, the one responsible for the latest CRISPR fiasco – has its authors complete a checklist to combat irreproducibility, à la Atul Gawande. And Nature Communications, among other journals, uses transparent peer review, in which authors can opt to have the reviews of their manuscript published anonymously alongside the completed paper. This practice "shows people how the paper evolved," said Koff "and keeps the reviewer and editor accountable. Did the reviewer identify the major problems with the paper? Because there are always major problems with a paper."
A startup aims to make medicines in space
Story by Big Think
On June 12, a SpaceX Falcon 9 rocket deployed 72 small satellites for customers — including the world’s first space factory.
The challenge: In 2019, pharma giant Merck revealed that an experiment on the International Space Station had shown how to make its blockbuster cancer drug Keytruda more stable. That meant it could now be administered via a shot rather than through an IV infusion.
The key to the discovery was the fact that particles behave differently when freed from the force of gravity — seeing how its drug crystalized in microgravity helped Merck figure out how to tweak its manufacturing process on Earth to produce the more stable version.
Microgravity research could potentially lead to many more discoveries like this one, or even the development of brand-new drugs, but ISS astronauts only have so much time for commercial experiments.
“There are many high-performance products that are only possible to make in zero-gravity, which is a manufacturing capability that cannot be replicated in any factory on Earth.”-- Will Bruey.
The only options for accessing microgravity (or free fall) outside of orbit, meanwhile, are parabolic airplane flights and drop towers, and those are only useful for experiments that require less than a minute in microgravity — Merck’s ISS experiment took 18 days.
The idea: In 2021, California startup Varda Space Industries announced its intention to build the world’s first space factory, to manufacture not only pharmaceuticals but other products that could benefit from being made in microgravity, such as semiconductors and fiber optic cables.
This factory would consist of a commercial satellite platform attached to two Varda-made modules. One module would contain equipment capable of autonomously manufacturing a product. The other would be a reentry capsule to bring the finished goods back to Earth.
“There are many high-performance products that are only possible to make in zero-gravity, which is a manufacturing capability that cannot be replicated in any factory on Earth,” said CEO Will Bruey, who’d previously developed and flown spacecraft for SpaceX.
“We have a team stacked with aerospace talent in the prime of their careers, focused on getting working hardware to orbit as quickly as possible,” he continued.
“[Pharmaceuticals] are the most valuable chemicals per unit mass. And they also have a large market on Earth.” -- Will Bruey, CEO of Varda Space.
What’s new? At the time, Varda said it planned to launch its first space factory in 2023, and, in what feels like a first for a space startup, it has actually hit that ambitious launch schedule.
“We have ACQUISITION OF SIGNAL,” the startup tweeted soon after the Falcon 9 launch on June 12. “The world’s first space factory’s solar panels have found the sun and it’s beginning to de-tumble.”
During the satellite’s first week in space, Varda will focus on testing its systems to make sure everything works as hoped. The second week will be dedicated to heating and cooling the old HIV-AIDS drug ritonavir repeatedly to study how its particles crystalize in microgravity.
After about a month in space, Varda will attempt to bring its first space factory back to Earth, sending it through the atmosphere at hypersonic speeds and then using a parachute system to safely land at the Department of Defense’s Utah Test and Training Range.
Looking ahead: Ultimately, Varda’s space factories could end up serving dual purposes as manufacturing facilities and hypersonic testbeds — the Air Force has already awarded the startup a contract to use its next reentry capsule to test hardware for hypersonic missiles.
But as for manufacturing other types of goods, Varda plans to stick with drugs for now.
“[Pharmaceuticals] are the most valuable chemicals per unit mass,” Bruey told CNN. “And they also have a large market on Earth.”
“You’re not going to see Varda do anything other than pharmaceuticals for the next minimum of six, seven years,” added Delian Asparouhov, Varda’s co-founder and president.
Genes that protect health with Dr. Nir Barzilai
In today’s podcast episode, I talk with Nir Barzilai, a geroscientist, which means he studies the biology of aging. Barzilai directs the Institute for Aging Research at the Albert Einstein College of Medicine.
My first question for Dr. Barzilai was: why do we age? And is there anything to be done about it? His answers were encouraging. We can’t live forever, but we have some control over the process, as he argues in his book, Age Later.
Dr. Barzilai told me that centenarians differ from the rest of us because they have unique gene mutations that help them stay healthy longer. For most of us, the words “gene mutations” spell trouble - we associate these words with cancer or neurodegenerative diseases, but apparently not all mutations are bad.
Listen on Apple | Listen on Spotify | Listen on Stitcher | Listen on Amazon | Listen on Google
Centenarians may have essentially won the genetic lottery, but that doesn’t mean the rest of us are predestined to have a specific lifespan and health span, or the amount of time spent living productively and enjoyably. “Aging is a mother of all diseases,” Dr. Barzilai told me. And as a disease, it can be targeted by therapeutics. Dr. Barzilai’s team is already running clinical trials on such therapeutics — and the results are promising.
More about Dr. Barzilai: He is scientific director of AFAR, American Federation for Aging Research. As part of his work, Dr. Barzilai studies families of centenarians and their genetics to learn how the rest of us can learn and benefit from their super-aging. He also organizing a clinical trial to test a specific drug that may slow aging.
Show Links
Age Later: Health Span, Life Span, and the New Science of Longevity https://www.amazon.com/Age-Later-Healthiest-Sharpest-Centenarians/dp/1250230853
American Federation for Aging Research https://www.afar.org
https://www.afar.org/nir-barzilai
https://www.einsteinmed.edu/faculty/484/nir-barzilai/
Metformin as a Tool to Target Aging
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5943638/
Benefits of Metformin in Attenuating the Hallmarks of Aging https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7347426/
The Longevity Genes Project https://www.einsteinmed.edu/centers/aging/longevity-genes-project/
Lina Zeldovich has written about science, medicine and technology for Popular Science, Smithsonian, National Geographic, Scientific American, Reader’s Digest, the New York Times and other major national and international publications. A Columbia J-School alumna, she has won several awards for her stories, including the ASJA Crisis Coverage Award for Covid reporting, and has been a contributing editor at Nautilus Magazine. In 2021, Zeldovich released her first book, The Other Dark Matter, published by the University of Chicago Press, about the science and business of turning waste into wealth and health. You can find her on http://linazeldovich.com/ and @linazeldovich.