Sloppy Science Happens More Than You Think
The media loves to tout scientific breakthroughs, and few are as toutable – and in turn, have been as touted – as CRISPR. This method of targeted DNA excision was discovered in bacteria, which use it as an adaptive immune system to combat reinfection with a previously encountered virus.
Shouldn't the editors at a Nature journal know better than to have published an incorrect paper in the first place?
It is cool on so many levels: not only is the basic function fascinating, reminding us that we still have more to discover about even simple organisms that we thought we knew so well, but the ability it grants us to remove and replace any DNA of interest has almost limitless applications in both the lab and the clinic. As if that didn't make it sexy enough, add in a bicoastal, male-female, very public and relatively ugly patent battle, and the CRISPR story is irresistible.
And then last summer, a bombshell dropped. The prestigious journal Nature Methods published a paper in which the authors claimed that CRISPR could cause many unintended mutations, rendering it unfit for clinical use. Havoc duly ensued; stocks in CRISPR-based companies plummeted. Thankfully, the authors of the offending paper were responsible, good scientists; they reassessed, then recanted. Their attention- and headline- grabbing results were wrong, and they admitted as much, leading Nature Methods to formally retract the paper this spring.
How did this happen? Shouldn't the editors at a Nature journal know better than to have published this in the first place?
Alas, high-profile scientific journals publish misleading and downright false results fairly regularly. Some errors are unavoidable – that's how the scientific method works. Hypotheses and conclusions will invariably be overturned as new data becomes available and new technologies are developed that allow for deeper and deeper studies. That's supposed to happen. But that's not what we're talking about here. Nor are we talking about obvious offenses like outright plagiarism. We're talking about mistakes that are avoidable, and that still have serious ramifications.
The cultures of both industry and academia promote research that is poorly designed and even more poorly analyzed.
Two parties are responsible for a scientific publication, and thus two parties bear the blame when things go awry: the scientists who perform and submit the work, and the journals who publish it. Unfortunately, both are incentivized for speedy and flashy publications, and not necessarily for correct publications. It is hardly a surprise, then, that we end up with papers that are speedy and flashy – and not necessarily correct.
"Scientists don't lie and submit falsified data," said Andy Koff, a professor of Molecular Biology at Sloan Kettering Institute, the basic research arm of Memorial Sloan Kettering Cancer Center. Richard Harris, who wrote the book on scientific misconduct running the gamut from unconscious bias and ignorance to more malicious fraudulence, largely concurs (full disclosure: I reviewed the book here). "Scientists want to do good science and want to be recognized as such," he said. But even so, the cultures of both industry and academia promote research that is poorly designed and even more poorly analyzed. In Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Millions, Harris describes how scientists must constantly publish in order to maintain their reputations and positions, to get grants and tenure and students. "They are disincentivized from doing that last extra experiment to prove their results," he said; it could prove too risky if it could cost them a publication.
Ivan Oransky and Adam Marcus founded Retraction Watch, a blog that tracks the retraction of scientific papers, in 2010. Oransky pointed out that blinded peer review – the pride and joy of the scientific publishing enterprise – is a large part of the problem. "Pre-publication peer review is still important, but we can't treat it like the only check on the system. Papers are being reviewed by non-experts, and reviewers are asked to review papers only tangentially related to their field. Moreover, most peer reviewers don't look at the underlying or raw data, even when it is available. How then can they tell if the analysis is flawed or the data is accurate?" he wondered.
Mistaken publications also erode the public's opinion of legitimate science, which is problematic since that opinion isn't especially high to begin with.
Koff agreed that anonymous peer review is valuable, but severely flawed. "Blinded review forces a collective view of importance," he said. "If an article disagrees with the reviewer's worldview, the article gets rejected or forced to adhere to that worldview – even if that means pushing the data someplace it shouldn't necessarily go." We have lost the scientific principle behind review, he thinks, which was to critically analyze a paper. But instead of challenging fundamental assumptions within a paper, reviewers now tend to just ask for more and more supplementary data. And don't get him started on editors. "Editors are supposed to arbitrate between reviewers and writers and they have completely abdicated this responsibility, at every journal. They do not judge, and that's a real failing."
Harris laments the wasted time, effort, and resources that result when erroneous ideas take hold in a field, not to mention lives lost when drug discovery is predicated on basic science findings that end up being wrong. "When no one takes the time, care, and money to reproduce things, science isn't stopping – but it is slowing down," he noted. Mistaken publications also erode the public's opinion of legitimate science, which is problematic since that opinion isn't especially high to begin with.
Scientists and publishers don't only cause the problem, though – they may also provide the solution. Both camps are increasingly recognizing and dealing with the crisis. The self-proclaimed "data thugs" Nick Brown and James Heathers use pretty basic arithmetic to reveal statistical errors in papers. The microbiologist Elisabeth Bik scans the scientific literature for problematic images "in her free time." The psychologist Brian Nosek founded the Center for Open Science, a non-profit organization dedicated to promoting openness, integrity, and reproducibility in scientific research. The Nature family of journals – yes, the one responsible for the latest CRISPR fiasco – has its authors complete a checklist to combat irreproducibility, à la Atul Gawande. And Nature Communications, among other journals, uses transparent peer review, in which authors can opt to have the reviews of their manuscript published anonymously alongside the completed paper. This practice "shows people how the paper evolved," said Koff "and keeps the reviewer and editor accountable. Did the reviewer identify the major problems with the paper? Because there are always major problems with a paper."
Indigenous wisdom plus honeypot ants could provide new antibiotics
For generations, the Indigenous Tjupan people of Australia enjoyed the sweet treat of honey made by honeypot ants. As a favorite pastime, entire families would go searching for the underground colonies, first spotting a worker ant and then tracing it to its home. The ants, which belong to the species called Camponotus inflatus, usually build their subterranean homes near the mulga trees, Acacia aneura. Having traced an ant to its tree, it would be the women who carefully dug a pit next to a colony, cautious not to destroy the entire structure. Once the ant chambers were exposed, the women would harvest a small amount to avoid devastating the colony’s stocks—and the family would share the treat.
The Tjupan people also knew that the honey had antimicrobial properties. “You could use it for a sore throat,” says Danny Ulrich, a member of the Tjupan nation. “You could also use it topically, on cuts and things like that.”
These hunts have become rarer, as many of the Tjupan people have moved away and, up until now, the exact antimicrobial properties of the ant honey remained unknown. But recently, scientists Andrew Dong and Kenya Fernandes from the University of Sydney, joined Ulrich, who runs the Honeypot Ants tours in Kalgoorlie, a city in Western Australia, on a honey-gathering expedition. Afterwards, they ran a series of experiments analyzing the honey’s antimicrobial activity—and confirmed that the Indigenous wisdom was true. The honey was effective against Staphylococcus aureus, a common pathogen responsible for sore throats, skin infections like boils and sores, and also sepsis, which can result in death. Moreover, the honey also worked against two species of fungi, Cryptococcus and Aspergillus, which can be pathogenic to humans, especially those with suppressed immune systems.
In the era of growing antibiotic resistance and the rising threat of pathogenic fungi, these findings may help scientists identify and make new antimicrobial compounds. “Natural products have been honed over thousands and millions of years by nature and evolution,” says Fernandes. “And some of them have complex and intricate properties that make them really important as potential new antibiotics. “
In an era of growing resistance to antibiotics and new threats of fungi infections, the latest findings about honeypot ants are helping scientists identify new antimicrobial drugs.
Danny Ulrich
Bee honey is also known for its antimicrobial properties, but bees produce it very differently than the ants. Bees collect nectar from flowers, which they regurgitate at the hive and pack into the hexagonal honeycombs they build for storage. As they do so, they also add into the mix an enzyme called glucose oxidase produced by their glands. The enzyme converts atmospheric oxygen into hydrogen peroxide, a reactive molecule that destroys bacteria and acts as a natural preservative. After the bees pack the honey into the honeycombs, they fan it with their wings to evaporate the water. Once a honeycomb is full, the bees put a beeswax cover on it, where it stays well-preserved thanks to the enzymatic action, until the bees need it.
Less is known about the chemistry of ants’ honey-making. Similarly to bees, they collect nectar. They also collect the sweet sap of the mulga tree. Additionally, they also “milk” the aphids—small sap-sucking insects that live on the tree. When ants tickle the aphids with their antennae, the latter release a sweet substance, which the former also transfer to their colonies. That’s where the honey management difference becomes really pronounced. The ants don’t build any kind of structures to store their honey. Instead, they store it in themselves.
The workers feed their harvest to their fellow ants called repletes, stuffing them up to the point that their swollen bellies outgrow the ants themselves, looking like amber-colored honeypots—hence the name. Because of their size, repletes don’t move, but hang down from the chamber’s ceiling, acting as living feedstocks. When food becomes scarce, they regurgitate their reserves to their colony’s brethren. It’s not clear whether the repletes die afterwards or can be restuffed again. “That's a good question,” Dong says. “After they've been stretched, they can't really return to exactly the same shape.”
These replete ants are the “treat” the Tjupan women dug for. Once they saw the round-belly ants inside the chambers, they would reach in carefully and get a few scoops of them. “You see a lot of honeypot ants just hanging on the roof of the little openings,” says Ulrich’s mother, Edie Ulrich. The women would share the ants with family members who would eat them one by one. “They're very delicate,” shares Edie Ulrich—you have to take them out carefully, so they don’t accidentally pop and become a wasted resource. “Because you’d lose all this precious honey.”
Dong stumbled upon the honeypot ants phenomenon because he was interested in Indigenous foods and went on Ulrich’s tour. He quickly became fascinated with the insects and their role in the Indigenous culture. “The honeypot ants are culturally revered by the Indigenous people,” he says. Eventually he decided to test out the honey’s medicinal qualities.
The researchers were surprised to see that even the smallest, eight percent concentration of honey was able to arrest the growth of S. aureus.
To do this, the two scientists first diluted the ant honey with water. “We used something called doubling dilutions, which means that we made 32 percent dilutions, and then we halve that to 16 percent and then we half that to eight percent,” explains Fernandes. The goal was to obtain as much results as possible with the meager honey they had. “We had very, very little of the honeypot ant honey so we wanted to maximize the spectrum of results we can get without wasting too much of the sample.”
After that, the researchers grew different microbes inside a nutrient rich broth. They added the broth to the different honey dilutions and incubated the mixes for a day or two at the temperature favorable to the germs’ growth. If the resulting solution turned turbid, it was a sign that the bugs proliferated. If it stayed clear, it meant that the honey destroyed them. The researchers were surprised to see that even the smallest, eight percent concentration of honey was able to arrest the growth of S. aureus. “It was really quite amazing,” Fernandes says. “Eight milliliters of honey in 92 milliliters of water is a really tiny amount of honey compared to the amount of water.”
Similar to bee honey, the ants’ honey exhibited some peroxide antimicrobial activity, researchers found, but given how little peroxide was in the solution, they think the honey also kills germs by a different mechanism. “When we measured, we found that [the solution] did have some hydrogen peroxide, but it didn't have as much of it as we would expect based on how active it was,” Fernandes says. “Whether this hydrogen peroxide also comes from glucose oxidase or whether it's produced by another source, we don't really know,” she adds. The research team does have some hypotheses about the identity of this other germ-killing agent. “We think it is most likely some kind of antimicrobial peptide that is actually coming from the ant itself.”
The honey also has a very strong activity against the two types of fungi, Cryptococcus and Aspergillus. Both fungi are associated with trees and decaying leaves, as well as in the soils where ants live, so the insects likely have evolved some natural defense compounds, which end up inside the honey.
It wouldn’t be the first time when modern medicines take their origin from the natural world or from the indigenous people’s knowledge. The bark of the cinchona tree native to South America contains quinine, a substance that treats malaria. The Indigenous people of the Andes used the bark to quell fever and chills for generations, and when Europeans began to fall ill with malaria in the Amazon rainforest, they learned to use that medicine from the Andean people.
The wonder drug aspirin similarly takes its origin from a bark of a tree—in this case a willow.
Even some anticancer compounds originated from nature. A chemotherapy drug called Paclitaxel, was originally extracted from the Pacific yew trees, Taxus brevifolia. The samples of the Pacific yew bark were first collected in 1962 by researchers from the United States Department of Agriculture who were looking for natural compounds that might have anti-tumor activity. In December 1992, the FDA approved Paclitaxel (brand name Taxol) for the treatment of ovarian cancer and two years later for breast cancer.
In the era when the world is struggling to find new medicines fast enough to subvert a fungal or bacterial pandemic, these discoveries can pave the way to new therapeutics. “I think it's really important to listen to indigenous cultures and to take their knowledge because they have been using these sources for a really, really long time,” Fernandes says. Now we know it works, so science can elucidate the molecular mechanisms behind it, she adds. “And maybe it can even provide a lead for us to develop some kind of new treatments in the future.”
Lina Zeldovich has written about science, medicine and technology for Popular Science, Smithsonian, National Geographic, Scientific American, Reader’s Digest, the New York Times and other major national and international publications. A Columbia J-School alumna, she has won several awards for her stories, including the ASJA Crisis Coverage Award for Covid reporting, and has been a contributing editor at Nautilus Magazine. In 2021, Zeldovich released her first book, The Other Dark Matter, published by the University of Chicago Press, about the science and business of turning waste into wealth and health. You can find her on http://linazeldovich.com/ and @linazeldovich.
Blood Test Can Detect Lymphoma Cells Before a Tumor Grows Back
When David M. Kurtz was doing his clinical fellowship at Stanford University Medical Center in 2009, specializing in lymphoma treatments, he found himself grappling with a question no one could answer. A typical regimen for these blood cancers prescribed six cycles of chemotherapy, but no one knew why. "The number seemed to be drawn out of a hat," Kurtz says. Some patients felt much better after just two doses, but had to endure the toxic effects of the entire course. For some elderly patients, the side effects of chemo are so harsh, they alone can kill. Others appeared to be cancer-free on the CT scans after the requisite six but then succumbed to it months later.
"Anecdotally, one patient decided to stop therapy after one dose because he felt it was so toxic that he opted for hospice instead," says Kurtz, now an oncologist at the center. "Five years down the road, he was alive and well. For him, just one dose was enough." Others would return for their one-year check up and find that their tumors grew back. Kurtz felt that while CT scans and MRIs were powerful tools, they weren't perfect ones. They couldn't tell him if there were any cancer cells left, stealthily waiting to germinate again. The scans only showed the tumor once it was back.
Blood cancers claim about 68,000 people a year, with a new diagnosis made about every three minutes, according to the Leukemia Research Foundation. For patients with B-cell lymphoma, which Kurtz focuses on, the survival chances are better than for some others. About 60 percent are cured, but the remaining 40 percent will relapse—possibly because they will have a negative CT scan, but still harbor malignant cells. "You can't see this on imaging," says Michael Green, who also treats blood cancers at University of Texas MD Anderson Medical Center.
The new blood test is sensitive enough to spot one cancerous perpetrator amongst one million other DNA molecules.
Kurtz wanted a better diagnostic tool, so he started working on a blood test that could capture the circulating tumor DNA or ctDNA. For that, he needed to identify the specific mutations typical for B-cell lymphomas. Working together with another fellow PhD student Jake Chabon, Kurtz finally zeroed-in on the tumor's genetic "appearance" in 2017—a pair of specific mutations sitting in close proximity to each other—a rare and telling sign. The human genome contains about 3 billion base pairs of nucleotides—molecules that compose genes—and in case of the B-cell lymphoma cells these two mutations were only a few base pairs apart. "That was the moment when the light bulb went on," Kurtz says.
The duo formed a company named Foresight Diagnostics, focusing on taking the blood test to the clinic. But knowing the tumor's mutational signature was only half the process. The other was fishing the tumor's DNA out of patients' bloodstream that contains millions of other DNA molecules, explains Chabon, now Foresight's CEO. It would be like looking for an escaped criminal in a large crowd. Kurtz and Chabon solved the problem by taking the tumor's "mug shot" first. Doctors would take the biopsy pre-treatment and sequence the tumor, as if taking the criminal's photo. After treatments, they would match the "mug shot" to all DNA molecules derived from the patient's blood sample to see if any molecular criminals managed to escape the chemo.
Foresight isn't the only company working on blood-based tumor detection tests, which are dubbed liquid biopsies—other companies such as Natera or ArcherDx developed their own. But in a recent study, the Foresight team showed that their method is significantly more sensitive in "fishing out" the cancer molecules than existing tests. Chabon says that this test can detect circulating tumor DNA in concentrations that are nearly 100 times lower than other methods. Put another way, it's sensitive enough to spot one cancerous perpetrator amongst one million other DNA molecules.
They also aim to extend their test to detect other malignancies such as lung, breast or colorectal cancers.
"It increases the sensitivity of detection and really catches most patients who are going to progress," says Green, the University of Texas oncologist who wasn't involved in the study, but is familiar with the method. It would also allow monitoring patients during treatment and making better-informed decisions about which therapy regimens would be most effective. "It's a minimally invasive test," Green says, and "it gives you a very high confidence about what's going on."
Having shown that the test works well, Kurtz and Chabon are planning a new trial in which oncologists would rely on their method to decide when to stop or continue chemo. They also aim to extend their test to detect other malignancies such as lung, breast or colorectal cancers. The latest genome sequencing technologies have sequenced and catalogued over 2,500 different tumor specimens and the Foresight team is analyzing this data, says Chabon, which gives the team the opportunity to create more molecular "mug shots."
The team hopes that that their blood cancer test will become available to patients within about five years, making doctors' job easier, and not only at the biological level. "When I tell patients, "good news, your cancer is in remission', they ask me, 'does it mean I'm cured?'" Kurtz says. "Right now I can't answer this question because I don't know—but I would like to." His company's test, he hopes, will enable him to reply with certainty. He'd very much like to have the power of that foresight.
This article is republished from our archives to coincide with Blood Cancer Awareness Month, which highlights progress in cancer diagnostics and treatment.
Lina Zeldovich has written about science, medicine and technology for Popular Science, Smithsonian, National Geographic, Scientific American, Reader’s Digest, the New York Times and other major national and international publications. A Columbia J-School alumna, she has won several awards for her stories, including the ASJA Crisis Coverage Award for Covid reporting, and has been a contributing editor at Nautilus Magazine. In 2021, Zeldovich released her first book, The Other Dark Matter, published by the University of Chicago Press, about the science and business of turning waste into wealth and health. You can find her on http://linazeldovich.com/ and @linazeldovich.