Sloppy Science Happens More Than You Think
Manipulating DNA through gene editing.
The media loves to tout scientific breakthroughs, and few are as toutable – and in turn, have been as touted – as CRISPR. This method of targeted DNA excision was discovered in bacteria, which use it as an adaptive immune system to combat reinfection with a previously encountered virus.
Shouldn't the editors at a Nature journal know better than to have published an incorrect paper in the first place?
It is cool on so many levels: not only is the basic function fascinating, reminding us that we still have more to discover about even simple organisms that we thought we knew so well, but the ability it grants us to remove and replace any DNA of interest has almost limitless applications in both the lab and the clinic. As if that didn't make it sexy enough, add in a bicoastal, male-female, very public and relatively ugly patent battle, and the CRISPR story is irresistible.
And then last summer, a bombshell dropped. The prestigious journal Nature Methods published a paper in which the authors claimed that CRISPR could cause many unintended mutations, rendering it unfit for clinical use. Havoc duly ensued; stocks in CRISPR-based companies plummeted. Thankfully, the authors of the offending paper were responsible, good scientists; they reassessed, then recanted. Their attention- and headline- grabbing results were wrong, and they admitted as much, leading Nature Methods to formally retract the paper this spring.
How did this happen? Shouldn't the editors at a Nature journal know better than to have published this in the first place?
Alas, high-profile scientific journals publish misleading and downright false results fairly regularly. Some errors are unavoidable – that's how the scientific method works. Hypotheses and conclusions will invariably be overturned as new data becomes available and new technologies are developed that allow for deeper and deeper studies. That's supposed to happen. But that's not what we're talking about here. Nor are we talking about obvious offenses like outright plagiarism. We're talking about mistakes that are avoidable, and that still have serious ramifications.
The cultures of both industry and academia promote research that is poorly designed and even more poorly analyzed.
Two parties are responsible for a scientific publication, and thus two parties bear the blame when things go awry: the scientists who perform and submit the work, and the journals who publish it. Unfortunately, both are incentivized for speedy and flashy publications, and not necessarily for correct publications. It is hardly a surprise, then, that we end up with papers that are speedy and flashy – and not necessarily correct.
"Scientists don't lie and submit falsified data," said Andy Koff, a professor of Molecular Biology at Sloan Kettering Institute, the basic research arm of Memorial Sloan Kettering Cancer Center. Richard Harris, who wrote the book on scientific misconduct running the gamut from unconscious bias and ignorance to more malicious fraudulence, largely concurs (full disclosure: I reviewed the book here). "Scientists want to do good science and want to be recognized as such," he said. But even so, the cultures of both industry and academia promote research that is poorly designed and even more poorly analyzed. In Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Millions, Harris describes how scientists must constantly publish in order to maintain their reputations and positions, to get grants and tenure and students. "They are disincentivized from doing that last extra experiment to prove their results," he said; it could prove too risky if it could cost them a publication.
Ivan Oransky and Adam Marcus founded Retraction Watch, a blog that tracks the retraction of scientific papers, in 2010. Oransky pointed out that blinded peer review – the pride and joy of the scientific publishing enterprise – is a large part of the problem. "Pre-publication peer review is still important, but we can't treat it like the only check on the system. Papers are being reviewed by non-experts, and reviewers are asked to review papers only tangentially related to their field. Moreover, most peer reviewers don't look at the underlying or raw data, even when it is available. How then can they tell if the analysis is flawed or the data is accurate?" he wondered.
Mistaken publications also erode the public's opinion of legitimate science, which is problematic since that opinion isn't especially high to begin with.
Koff agreed that anonymous peer review is valuable, but severely flawed. "Blinded review forces a collective view of importance," he said. "If an article disagrees with the reviewer's worldview, the article gets rejected or forced to adhere to that worldview – even if that means pushing the data someplace it shouldn't necessarily go." We have lost the scientific principle behind review, he thinks, which was to critically analyze a paper. But instead of challenging fundamental assumptions within a paper, reviewers now tend to just ask for more and more supplementary data. And don't get him started on editors. "Editors are supposed to arbitrate between reviewers and writers and they have completely abdicated this responsibility, at every journal. They do not judge, and that's a real failing."
Harris laments the wasted time, effort, and resources that result when erroneous ideas take hold in a field, not to mention lives lost when drug discovery is predicated on basic science findings that end up being wrong. "When no one takes the time, care, and money to reproduce things, science isn't stopping – but it is slowing down," he noted. Mistaken publications also erode the public's opinion of legitimate science, which is problematic since that opinion isn't especially high to begin with.
Scientists and publishers don't only cause the problem, though – they may also provide the solution. Both camps are increasingly recognizing and dealing with the crisis. The self-proclaimed "data thugs" Nick Brown and James Heathers use pretty basic arithmetic to reveal statistical errors in papers. The microbiologist Elisabeth Bik scans the scientific literature for problematic images "in her free time." The psychologist Brian Nosek founded the Center for Open Science, a non-profit organization dedicated to promoting openness, integrity, and reproducibility in scientific research. The Nature family of journals – yes, the one responsible for the latest CRISPR fiasco – has its authors complete a checklist to combat irreproducibility, à la Atul Gawande. And Nature Communications, among other journals, uses transparent peer review, in which authors can opt to have the reviews of their manuscript published anonymously alongside the completed paper. This practice "shows people how the paper evolved," said Koff "and keeps the reviewer and editor accountable. Did the reviewer identify the major problems with the paper? Because there are always major problems with a paper."
Chicken that is grown entirely in a laboratory, without harming a single bird, could be sold in supermarkets in the coming months. But critics say the doubts about lab-grown meat have not been appropriately explored.
Last November, when the U.S. Food and Drug Administration disclosed that chicken from a California firm called UPSIDE Foods did not raise safety concerns, it drily upended how humans have obtained animal protein for thousands of generations.
“The FDA is ready to work with additional firms developing cultured animal cell food and production processes to ensure their food is safe and lawful,” the agency said in a statement at the time.
Assuming UPSIDE obtains clearances from the U.S. Department of Agriculture, its chicken – grown entirely in a laboratory without harming a single bird – could be sold in supermarkets in the coming months.
“Ultimately, we want our products to be available everywhere meat is sold, including retail and food service channels,” a company spokesperson said. The upscale French restaurant Atelier Crenn in San Francisco will have UPSIDE chicken on its menu once it is approved, she added.
Known as lab-grown or cultured meat, a product such as UPSIDE’s is created using stem cells and other tissue obtained from a chicken, cow or other livestock. Those cells are then multiplied in a nutrient-dense environment, usually in conjunction with a “scaffold” of plant-based materials or gelatin to give them a familiar form, such as a chicken breast or a ribeye steak. A Dutch company called Mosa Meat claims it can produce 80,000 hamburgers derived from a cluster of tissue the size of a sesame seed.
Critics say the doubts about lab-grown meat and the possibility it could merge “Brave New World” with “The Jungle” and “Soylent Green” have not been appropriately explored.
That’s a far cry from when it took months of work to create the first lab-grown hamburger a decade ago. That minuscule patty – which did not contain any fat and was literally plucked from a Petri dish to go into a frying pan – cost about $325,000 to produce.
Just a decade later, an Israeli company called Future Meat said it can produce lab-grown meat for about $1.70 per pound. It plans to open a production facility in the U.S. sometime in 2023 and distribute its products under the brand name “Believer.”
Costs for production have sunk so low that researchers at Carnegie Mellon University in Pittsburgh expect sometime in early 2024 to produce lab-grown Wagyu steak to showcase the viability of growing high-end cuts of beef cheaply. The Carnegie Mellon team is producing its Wagyu using a consumer 3-D printer bought secondhand on eBay and modified to print the highly marbled flesh using a method developed by the university. The device costs $200 – about the same as a pound of Wagyu in the U.S. The initiative’s modest five-figure budget was successfully crowdfunded last year.
“The big cost is going to be the cells (which are being extracted by a cow somewhere in Pennsylvania), but otherwise printing doesn’t add much to the process,” said Rosalyn Abbott, a Carnegie Mellon assistant professor of bioengineering who is co-leader on the project. “But it adds value, unlike doing this with ground meat.”
Lab-Grown Meat’s Promise
Proponents of lab-grown meat say it will cut down on traditional agriculture, which has been a leading contributor to deforestation, water shortages and contaminated waterways from animal waste, as well as climate change.
An Oxford University study from 2011 concludes lab-grown meat could have greenhouse emissions 96 percent lower compared to traditionally raised livestock. Moreover, proponents of lab-grown meat claim that the suffering of animals would decline dramatically, as they would no longer need to be warehoused and slaughtered. A recently opened 26-story high-rise in China dedicated to the raising and slaughtering of pigs illustrates the current plight of livestock in stark terms.
Scientists may even learn how to tweak lab-grown meat to make it more nutritious. Natural red meat is high in saturated fat and, if it’s eaten too often, can lead to chronic diseases. In lab versions, the saturated fat could be swapped for healthier, omega-3 fatty acids.
But critics say the doubts about lab-grown meat and the possibility it could merge “Brave New World” with “The Jungle” and “Soylent Green” have not been appropriately explored.
A Slippery Slope?
Some academics who have studied the moral and ethical issues surrounding lab-grown meat believe it will have a tough path ahead gaining acceptance by consumers. Should it actually succeed in gaining acceptance, many ethical questions must be answered.
“People might be interested” in lab-grown meat, perhaps as a curiosity, said Carlos Alvaro, an associate professor of philosophy at the New York City College of Technology, part of the City University of New York. But the allure of traditionally sourced meat has been baked – or perhaps grilled – into people’s minds for so long that they may not want to make the switch. Plant-based meat provides a recent example of the uphill battle involved in changing old food habits, with Beyond Meat’s stock prices dipping nearly 80 percent in 2022.
"There are many studies showing that people don’t really care about the environment (to that extent)," Alvaro said. "So I don’t know how you would convince people to do this because of the environment.”
“From my research, I understand that the taste (of lab-grown meat) is not quite there,” Alvaro said, noting that the amino acids, sugars and other nutrients required to grow cultivated meat do not mimic what livestock are fed. He also observed that the multiplication of cells as part of the process “really mimic cancer cells” in the way they grow, another off-putting thought for would-be consumers of the product.
Alvaro is also convinced the public will not buy into any argument that lab-grown meat is more environmentally friendly.
“If people care about the environment, they either try and consume considerably less meat and other animal products, or they go vegan or vegetarian,” he said. “But there are many studies showing that people don’t really care about the environment (to that extent). So I don’t know how you would convince people to do this because of the environment.”
Ben Bramble, a professor at Australian National University who previously held posts at Princeton and Trinity College in Ireland, takes a slightly different tack. He noted that “if lab-grown meat becomes cheaper, healthier, or tastier than regular meat, there will be a large market for it. If it becomes all of these things, it will dominate the market.”
However, Bramble has misgivings about that occurring. He believes a smooth transition from traditionally sourced meat to a lab-grown version would allow humans to elide over the decades of animal cruelty perpetrated by large-scale agriculture, without fully reckoning with and learning from this injustice.
“My fear is that if we all switch over to lab-grown meat because it has become cheaper, healthier, or tastier than regular meat, we might never come to realize what we have done, and the terrible things we are capable of,” he said. “This would be a catastrophe.”
Bramble’s writings about cultured meat also raise some serious moral conundrums. If, for example, animal meat may be cultivated without killing animals, why not create products from human protein?
Actually, that’s already happened.
It occurred in 2019, when Orkan Telhan, a professor of fine arts at the University of Pennsylvania, collaborated with two scientists to create an art exhibit at the Philadelphia Museum of Art on the future of foodstuffs.
Although the exhibit included bioengineered bread and genetically modified salmon, it was an installation called “Ouroboros Steak” that drew the most attention. That was comprised of pieces of human flesh grown in a lab from cultivated cells and expired blood products obtained from online sources.
The exhibit was presented as four tiny morsels of red meat – shaped in patterns suggesting an ouroboros, a dragon eating its own tail. They were placed in tiny individual saucers atop a larger plate and placemat with a calico pattern, suggesting an item to order in a diner. The artwork drew international headlines – as well as condemnation for Telhan’s vision.
Telhan’s artwork is intended to critique the overarching assumption that lab-grown meat will eventually replace more traditional production methods, as well as the lack of transparency surrounding many processed foodstuffs. “They think that this problem (from industrial-scale agriculture) is going be solved by this new technology,” Telhan said. “I am critical (of) that perspective.”
Unlike Bramble, Telhan is not against lab-grown meat, so long as its producers are transparent about the sourcing of materials and its cultivation. But he believes that large-scale agricultural meat production – which dates back centuries – is not going to be replaced so quickly.
“We see this again and again with different industries, like algae-based fuels. A lot of companies were excited about this, and promoted it,” Telhan said. “And years later, we know these fuels work. But to be able to displace the oil industry means building the infrastructure to scale takes billions of dollars, and nobody has the patience or money to do it.”
Alvaro concurred on this point, which he believes is already weakened because a large swath of consumers aren’t concerned about environmental degradation.
“They’re going to have to sell this big, but in order to convince people to do so, they have to convince them to eat this product instead of regular meat,” Alvaro said.
Hidden Tweaks?
Moreover, if lab-based meat does obtain a significant market share, Telhan suggested companies may do things to the product – such as to genetically modify it to become more profitable – and never notify consumers. That is a particular concern in the U.S., where regulations regarding such modifications are vastly more relaxed than in the European Union.
“I think that they have really good objectives, and they aspire to good objectives,” Telhan said. “But the system itself doesn't really allow for that much transparency.”
No matter what the future holds, sometime next year Carnegie Mellon is expected to hold a press conference announcing it has produced a cut of the world’s most expensive beef with the help of a modified piece of consumer electronics. It will likely take place at around the same time UPSIDE chicken will be available for purchase in supermarkets and restaurants, pending the USDA’s approvals.
Abbott, the Carnegie Mellon professor, suggested the future event will be both informative and celebratory.
“I think Carnegie Mellon would have someone potentially cook it for us,” she said. “Like have a really good chef in New York City do it.”
In this week's Friday Five, breathing this way may cut down on anxiety, a fasting regimen that could make you sick, this type of job makes men more virile, 3D printed hearts could save your life, and the role of metformin in preventing dementia.
The Friday Five covers five stories in research that you may have missed this week. There are plenty of controversies and troubling ethical issues in science – and we get into many of them in our online magazine – but this news roundup focuses on scientific creativity and progress to give you a therapeutic dose of inspiration headed into the weekend.
Here are the promising studies covered in this week's Friday Five, featuring interviews with Dr. David Spiegel, associate chair of psychiatry and behavioral sciences at Stanford, and Dr. Filip Swirski, professor of medicine and cardiology at the Icahn School of Medicine at Mount Sinai.
Listen on Apple | Listen on Spotify | Listen on Stitcher | Listen on Amazon | Listen on Google
Here are the promising studies covered in this week's Friday Five, featuring interviews with Dr. David Spiegel, associate chair of psychiatry and behavioral sciences at Stanford, and Dr. Filip Swirski, professor of medicine and cardiology at the Icahn School of Medicine at Mount Sinai.
- Breathing this way cuts down on anxiety*
- Could your fasting regimen make you sick?
- This type of job makes men more virile
- 3D printed hearts could save your life
- Yet another potential benefit of metformin
* This video with Dr. Andrew Huberman of Stanford shows exactly how to do the breathing practice.