Sloppy Science Happens More Than You Think
The media loves to tout scientific breakthroughs, and few are as toutable – and in turn, have been as touted – as CRISPR. This method of targeted DNA excision was discovered in bacteria, which use it as an adaptive immune system to combat reinfection with a previously encountered virus.
Shouldn't the editors at a Nature journal know better than to have published an incorrect paper in the first place?
It is cool on so many levels: not only is the basic function fascinating, reminding us that we still have more to discover about even simple organisms that we thought we knew so well, but the ability it grants us to remove and replace any DNA of interest has almost limitless applications in both the lab and the clinic. As if that didn't make it sexy enough, add in a bicoastal, male-female, very public and relatively ugly patent battle, and the CRISPR story is irresistible.
And then last summer, a bombshell dropped. The prestigious journal Nature Methods published a paper in which the authors claimed that CRISPR could cause many unintended mutations, rendering it unfit for clinical use. Havoc duly ensued; stocks in CRISPR-based companies plummeted. Thankfully, the authors of the offending paper were responsible, good scientists; they reassessed, then recanted. Their attention- and headline- grabbing results were wrong, and they admitted as much, leading Nature Methods to formally retract the paper this spring.
How did this happen? Shouldn't the editors at a Nature journal know better than to have published this in the first place?
Alas, high-profile scientific journals publish misleading and downright false results fairly regularly. Some errors are unavoidable – that's how the scientific method works. Hypotheses and conclusions will invariably be overturned as new data becomes available and new technologies are developed that allow for deeper and deeper studies. That's supposed to happen. But that's not what we're talking about here. Nor are we talking about obvious offenses like outright plagiarism. We're talking about mistakes that are avoidable, and that still have serious ramifications.
The cultures of both industry and academia promote research that is poorly designed and even more poorly analyzed.
Two parties are responsible for a scientific publication, and thus two parties bear the blame when things go awry: the scientists who perform and submit the work, and the journals who publish it. Unfortunately, both are incentivized for speedy and flashy publications, and not necessarily for correct publications. It is hardly a surprise, then, that we end up with papers that are speedy and flashy – and not necessarily correct.
"Scientists don't lie and submit falsified data," said Andy Koff, a professor of Molecular Biology at Sloan Kettering Institute, the basic research arm of Memorial Sloan Kettering Cancer Center. Richard Harris, who wrote the book on scientific misconduct running the gamut from unconscious bias and ignorance to more malicious fraudulence, largely concurs (full disclosure: I reviewed the book here). "Scientists want to do good science and want to be recognized as such," he said. But even so, the cultures of both industry and academia promote research that is poorly designed and even more poorly analyzed. In Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Millions, Harris describes how scientists must constantly publish in order to maintain their reputations and positions, to get grants and tenure and students. "They are disincentivized from doing that last extra experiment to prove their results," he said; it could prove too risky if it could cost them a publication.
Ivan Oransky and Adam Marcus founded Retraction Watch, a blog that tracks the retraction of scientific papers, in 2010. Oransky pointed out that blinded peer review – the pride and joy of the scientific publishing enterprise – is a large part of the problem. "Pre-publication peer review is still important, but we can't treat it like the only check on the system. Papers are being reviewed by non-experts, and reviewers are asked to review papers only tangentially related to their field. Moreover, most peer reviewers don't look at the underlying or raw data, even when it is available. How then can they tell if the analysis is flawed or the data is accurate?" he wondered.
Mistaken publications also erode the public's opinion of legitimate science, which is problematic since that opinion isn't especially high to begin with.
Koff agreed that anonymous peer review is valuable, but severely flawed. "Blinded review forces a collective view of importance," he said. "If an article disagrees with the reviewer's worldview, the article gets rejected or forced to adhere to that worldview – even if that means pushing the data someplace it shouldn't necessarily go." We have lost the scientific principle behind review, he thinks, which was to critically analyze a paper. But instead of challenging fundamental assumptions within a paper, reviewers now tend to just ask for more and more supplementary data. And don't get him started on editors. "Editors are supposed to arbitrate between reviewers and writers and they have completely abdicated this responsibility, at every journal. They do not judge, and that's a real failing."
Harris laments the wasted time, effort, and resources that result when erroneous ideas take hold in a field, not to mention lives lost when drug discovery is predicated on basic science findings that end up being wrong. "When no one takes the time, care, and money to reproduce things, science isn't stopping – but it is slowing down," he noted. Mistaken publications also erode the public's opinion of legitimate science, which is problematic since that opinion isn't especially high to begin with.
Scientists and publishers don't only cause the problem, though – they may also provide the solution. Both camps are increasingly recognizing and dealing with the crisis. The self-proclaimed "data thugs" Nick Brown and James Heathers use pretty basic arithmetic to reveal statistical errors in papers. The microbiologist Elisabeth Bik scans the scientific literature for problematic images "in her free time." The psychologist Brian Nosek founded the Center for Open Science, a non-profit organization dedicated to promoting openness, integrity, and reproducibility in scientific research. The Nature family of journals – yes, the one responsible for the latest CRISPR fiasco – has its authors complete a checklist to combat irreproducibility, à la Atul Gawande. And Nature Communications, among other journals, uses transparent peer review, in which authors can opt to have the reviews of their manuscript published anonymously alongside the completed paper. This practice "shows people how the paper evolved," said Koff "and keeps the reviewer and editor accountable. Did the reviewer identify the major problems with the paper? Because there are always major problems with a paper."
The Friday Five: How to exercise for cancer prevention
The Friday Five covers five stories in research that you may have missed this week. There are plenty of controversies and troubling ethical issues in science – and we get into many of them in our online magazine – but this news roundup focuses on scientific creativity and progress to give you a therapeutic dose of inspiration headed into the weekend.
Listen on Apple | Listen on Spotify | Listen on Stitcher | Listen on Amazon | Listen on Google
Here are the promising studies covered in this week's Friday Five:
- How to exercise for cancer prevention
- A device that brings relief to back pain
- Ingredients for reducing Alzheimer's risk
- Is the world's oldest disease the fountain of youth?
- Scared of crossing bridges? Your phone can help
New approach to brain health is sparking memories
What if a few painless electrical zaps to your brain could help you recall names, perform better on Wordle or even ward off dementia?
This is where neuroscientists are going in efforts to stave off age-related memory loss as well as Alzheimer’s disease. Medications have shown limited effectiveness in reversing or managing loss of brain function so far. But new studies suggest that firing up an aging neural network with electrical or magnetic current might keep brains spry as we age.
Welcome to non-invasive brain stimulation (NIBS). No surgery or anesthesia is required. One day, a jolt in the morning with your own battery-operated kit could replace your wake-up coffee.
Scientists believe brain circuits tend to uncouple as we age. Since brain neurons communicate by exchanging electrical impulses with each other, the breakdown of these links and associations could be what causes the “senior moment”—when you can’t remember the name of the movie you just watched.
In 2019, Boston University researchers led by Robert Reinhart, director of the Cognitive and Clinical Neuroscience Laboratory, showed that memory loss in healthy older adults is likely caused by these disconnected brain networks. When Reinhart and his team stimulated two key areas of the brain with mild electrical current, they were able to bring the brains of older adult subjects back into sync — enough so that their ability to remember small differences between two images matched that of much younger subjects for at least 50 minutes after the testing stopped.
Reinhart wowed the neuroscience community once again this fall. His newer study in Nature Neuroscience presented 150 healthy participants, ages 65 to 88, who were able to recall more words on a given list after 20 minutes of low-intensity electrical stimulation sessions over four consecutive days. This amounted to a 50 to 65 percent boost in their recall.
Even Reinhart was surprised to discover the enhanced performance of his subjects lasted a full month when they were tested again later. Those who benefited most were the participants who were the most forgetful at the start.
An older person participates in Robert Reinhart's research on brain stimulation.
Robert Reinhart
Reinhart’s subjects only suffered normal age-related memory deficits, but NIBS has great potential to help people with cognitive impairment and dementia, too, says Krista Lanctôt, the Bernick Chair of Geriatric Psychopharmacology at Sunnybrook Health Sciences Center in Toronto. Plus, “it is remarkably safe,” she says.
Lanctôt was the senior author on a meta-analysis of brain stimulation studies published last year on people with mild cognitive impairment or later stages of Alzheimer’s disease. The review concluded that magnetic stimulation to the brain significantly improved the research participants’ neuropsychiatric symptoms, such as apathy and depression. The stimulation also enhanced global cognition, which includes memory, attention, executive function and more.
This is the frontier of neuroscience.
The two main forms of NIBS – and many questions surrounding them
There are two types of NIBS. They differ based on whether electrical or magnetic stimulation is used to create the electric field, the type of device that delivers the electrical current and the strength of the current.
Transcranial Current Brain Stimulation (tES) is an umbrella term for a group of techniques using low-wattage electrical currents to manipulate activity in the brain. The current is delivered to the scalp or forehead via electrodes attached to a nylon elastic cap or rubber headband.
Variations include how the current is delivered—in an alternating pattern or in a constant, direct mode, for instance. Tweaking frequency, potency or target brain area can produce different effects as well. Reinhart’s 2022 study demonstrated that low or high frequencies and alternating currents were uniquely tied to either short-term or long-term memory improvements.
Sessions may be 20 minutes per day over the course of several days or two weeks. “[The subject] may feel a tingling, warming, poking or itching sensation,” says Reinhart, which typically goes away within a minute.
The other main approach to NIBS is Transcranial Magnetic Simulation (TMS). It involves the use of an electromagnetic coil that is held or placed against the forehead or scalp to activate nerve cells in the brain through short pulses. The stimulation is stronger than tES but similar to a magnetic resonance imaging (MRI) scan.
The subject may feel a slight knocking or tapping on the head during a 20-to-60-minute session. Scalp discomfort and headaches are reported by some; in very rare cases, a seizure can occur.
No head-to-head trials have been conducted yet to evaluate the differences and effectiveness between electrical and magnetic current stimulation, notes Lanctôt, who is also a professor of psychiatry and pharmacology at the University of Toronto. Although TMS was approved by the FDA in 2008 to treat major depression, both techniques are considered experimental for the purpose of cognitive enhancement.
“One attractive feature of tES is that it’s inexpensive—one-fifth the price of magnetic stimulation,” Reinhart notes.
Don’t confuse either of these procedures with the horrors of electroconvulsive therapy (ECT) in the 1950s and ‘60s. ECT is a more powerful, riskier procedure used only as a last resort in treating severe mental illness today.
Clinical studies on NIBS remain scarce. Standardized parameters and measures for testing have not been developed. The high heterogeneity among the many existing small NIBS studies makes it difficult to draw general conclusions. Few of the studies have been replicated and inconsistencies abound.
Scientists are still lacking so much fundamental knowledge about the brain and how it works, says Reinhart. “We don’t know how information is represented in the brain or how it’s carried forward in time. It’s more complex than physics.”
Lanctôt’s meta-analysis showed improvements in global cognition from delivering the magnetic form of the stimulation to people with Alzheimer’s, and this finding was replicated inan analysis in the Journal of Prevention of Alzheimer’s Disease this fall. Neither meta-analysis found clear evidence that applying the electrical currents, was helpful for Alzheimer’s subjects, but Lanctôt suggests this might be merely because the sample size for tES was smaller compared to the groups that received TMS.
At the same time, London neuroscientist Marco Sandrini, senior lecturer in psychology at the University of Roehampton, critically reviewed a series of studies on the effects of tES on episodic memory. Often declining with age, episodic memory relates to recalling a person’s own experiences from the past. Sandrini’s review concluded that delivering tES to the prefrontal or temporoparietal cortices of the brain might enhance episodic memory in older adults with Alzheimer’s disease and amnesiac mild cognitive impairment (the predementia phase of Alzheimer’s when people start to have symptoms).
Researchers readily tick off studies needed to explore, clarify and validate existing NIBS data. What is the optimal stimulus session frequency, spacing and duration? How intense should the stimulus be and where should it be targeted for what effect? How might genetics or degree of brain impairment affect responsiveness? Would adjunct medication or cognitive training boost positive results? Could administering the stimulus while someone sleeps expedite memory consolidation?
Using MRI or another brain scan along with computational modeling of the current flow, a clinician could create a treatment that is customized to each person’s brain.
While Sandrini’s review reported improvements induced by tES in the recall or recognition of words and images, there is no evidence it will translate into improvements in daily activities. This is another question that will require more research and testing, Sandrini notes.
Scientists are still lacking so much fundamental knowledge about the brain and how it works, says Reinhart. “We don’t know how information is represented in the brain or how it’s carried forward in time. It’s more complex than physics.”
Where the science is headed
Learning how to apply precision medicine to NIBS is the next focus in advancing this technology, says Shankar Tumati, a post-doctoral fellow working with Lanctôt.
There is great variability in each person’s brain anatomy—the thickness of the skull, the brain’s unique folds, the amount of cerebrospinal fluid. All of these structural differences impact how electrical or magnetic stimulation is distributed in the brain and ultimately the effects.
Using MRI or another brain scan along with computational modeling of the current flow, a clinician could create a treatment that is customized to each person’s brain, from where to put the electrodes to determining the exact dose and duration of stimulation needed to achieve lasting results, Sandrini says.
Above all, most neuroscientists say that largescale research studies over long periods of time are necessary to confirm the safety and durability of this therapy for the purpose of boosting memory. Short of that, there can be no FDA approval or medical regulation for this clinical use.
Lanctôt urges people to seek out clinical NIBS trials in their area if they want to see the science advance. “That is how we’ll find the answers,” she says, predicting it will be 5 to 10 years to develop each additional clinical application of NIBS. Ultimately, she predicts that reigning in Alzheimer’s disease and mild cognitive impairment will require a multi-pronged approach that includes lifestyle and medications, too.
Sandrini believes that scientific efforts should focus on preventing or delaying Alzheimer’s. “We need to start intervention earlier—as soon as people start to complain about forgetting things,” he says. “Changes in the brain start 10 years before [there is a problem]. Once Alzheimer’s develops, it is too late.”