Researchers Behaving Badly: Known Frauds Are "the Tip of the Iceberg"
Last week, the whistleblowers in the Paolo Macchiarini affair at Sweden's Karolinska Institutet went on the record here to detail the retaliation they suffered for trying to expose a star surgeon's appalling research misconduct.
Scientific fraud of the type committed by Macchiarini is rare, but studies suggest that it's on the rise.
The whistleblowers had discovered that in six published papers, Macchiarini falsified data, lied about the condition of patients and circumvented ethical approvals. As a result, multiple patients suffered and died. But Karolinska turned a blind eye for years.
Scientific fraud of the type committed by Macchiarini is rare, but studies suggest that it's on the rise. Just this week, for example, Retraction Watch and STAT together broke the news that a Harvard Medical School cardiologist and stem cell researcher, Piero Anversa, falsified data in a whopping 31 papers, which now have to be retracted. Anversa had claimed that he could regenerate heart muscle by injecting bone marrow cells into damaged hearts, a result that no one has been able to duplicate.
A 2009 study published in the Public Library of Science (PLOS) found that about two percent of scientists admitted to committing fabrication, falsification or plagiarism in their work. That's a small number, but up to one third of scientists admit to committing "questionable research practices" that fall into a gray area between rigorous accuracy and outright fraud.
These dubious practices may include misrepresentations, research bias, and inaccurate interpretations of data. One common questionable research practice entails formulating a hypothesis after the research is done in order to claim a successful premise. Another highly questionable practice that can shape research is ghost-authoring by representatives of the pharmaceutical industry and other for-profit fields. Still another is gifting co-authorship to unqualified but powerful individuals who can advance one's career. Such practices can unfairly bolster a scientist's reputation and increase the likelihood of getting the work published.
The above percentages represent what scientists admit to doing themselves; when they evaluate the practices of their colleagues, the numbers jump dramatically. In a 2012 study published in the Journal of Research in Medical Sciences, researchers estimated that 14 percent of other scientists commit serious misconduct, while up to 72 percent engage in questionable practices. While these are only estimates, the problem is clearly not one of just a few bad apples.
In the PLOS study, Daniele Fanelli says that increasing evidence suggests the known frauds are "just the 'tip of the iceberg,' and that many cases are never discovered" because fraud is extremely hard to detect.
Essentially everyone wants to be associated with big breakthroughs, and they may overlook scientifically shaky foundations when a major advance is claimed.
In addition, it's likely that most cases of scientific misconduct go unreported because of the high price of whistleblowing. Those in the Macchiarini case showed extraordinary persistence in their multi-year campaign to stop his deadly trachea implants, while suffering serious damage to their careers. Such heroic efforts to unmask fraud are probably rare.
To make matters worse, there are numerous players in the scientific world who may be complicit in either committing misconduct or covering it up. These include not only primary researchers but co-authors, institutional executives, journal editors, and industry leaders. Essentially everyone wants to be associated with big breakthroughs, and they may overlook scientifically shaky foundations when a major advance is claimed.
Another part of the problem is that it's rare for students in science and medicine to receive an education in ethics. And studies have shown that older, more experienced and possibly jaded researchers are more likely to fudge results than their younger, more idealistic colleagues.
So, given the steep price that individuals and institutions pay for scientific misconduct, what compels them to go down that road in the first place? According to the JRMS study, individuals face intense pressures to publish and to attract grant money in order to secure teaching positions at universities. Once they have acquired positions, the pressure is on to keep the grants and publishing credits coming in order to obtain tenure, be appointed to positions on boards, and recruit flocks of graduate students to assist in research. And not to be underestimated is the human ego.
Paolo Macchiarini is an especially vivid example of a scientist seeking not only fortune, but fame. He liberally (and falsely) claimed powerful politicians and celebrities, even the Pope, as patients or admirers. He may be an extreme example, but we live in an age of celebrity scientists who bring huge amounts of grant money and high prestige to the institutions that employ them.
The media plays a significant role in both glorifying stars and unmasking frauds. In the Macchiarini scandal, the media first lifted him up, as in NBC's laudatory documentary, "A Leap of Faith," which painted him as a kind of miracle-worker, and then brought him down, as in the January 2016 documentary, "The Experiments," which chronicled the agonizing death of one of his patients.
Institutions can also play a crucial role in scientific fraud by putting more emphasis on the number and frequency of papers published than on their quality. The whole course of a scientist's career is profoundly affected by something called the h-index. This is a number based on both the frequency of papers published and how many times the papers are cited by other researchers. Raising one's ranking on the h-index becomes an overriding goal, sometimes eclipsing the kind of patient, time-consuming research that leads to true breakthroughs based on reliable results.
Universities also create a high-pressured environment that encourages scientists to cut corners. They, too, place a heavy emphasis on attracting large monetary grants and accruing fame and prestige. This can lead them, just as it led Karolinska, to protect a star scientist's sloppy or questionable research. According to Dr. Andrew Rosenberg, who is director of the Center for Science and Democracy at the U.S.-based Union of Concerned Scientists, "Karolinska defended its investment in an individual as opposed to the long-term health of the institution. People were dying, and they should have outsourced the investigation from the very beginning."
Having institutions investigate their own practices is a conflict of interest from the get-go, says Rosenberg.
Scientists, universities, and research institutions are also not immune to fads. "Hot" subjects attract grant money and confer prestige, incentivizing scientists to shift their research priorities in a direction that garners more grants. This can mean neglecting the scientist's true area of expertise and interests in favor of a subject that's more likely to attract grant money. In Macchiarini's case, he was allegedly at the forefront of the currently sexy field of regenerative medicine -- a field in which Karolinska was making a huge investment.
The relative scarcity of resources intensifies the already significant pressure on scientists. They may want to publish results rapidly, since they face many competitors for limited grant money, academic positions, students, and influence. The scarcity means that a great many researchers will fail while only a few succeed. Once again, the temptation may be to rush research and to show it in the most positive light possible, even if it means fudging or exaggerating results.
Though the pressures facing scientists are very real, the problem of misconduct is not inevitable.
Intense competition can have a perverse effect on researchers, according to a 2007 study in the journal Science of Engineering and Ethics. Not only does it place undue pressure on scientists to succeed, it frequently leads to the withholding of information from colleagues, which undermines a system in which new discoveries build on the previous work of others. Researchers may feel compelled to withhold their results because of the pressure to be the first to publish. The study's authors propose that more investment in basic research from governments could alleviate some of these competitive pressures.
Scientific journals, although they play a part in publishing flawed science, can't be expected to investigate cases of suspected fraud, says the German science blogger Leonid Schneider. Schneider's writings helped to expose the Macchiarini affair.
"They just basically wait for someone to retract problematic papers," he says.
He also notes that, while American scientists can go to the Office of Research Integrity to report misconduct, whistleblowers in Europe have no external authority to whom they can appeal to investigate cases of fraud.
"They have to go to their employer, who has a vested interest in covering up cases of misconduct," he says.
Science is increasingly international. Major studies can include collaborators from several different countries, and he suggests there should be an international body accessible to all researchers that will investigate suspected fraud.
Ultimately, says Rosenberg, the scientific system must incorporate trust. "You trust co-authors when you write a paper, and peer reviewers at journals trust that scientists at research institutions like Karolinska are acting with integrity."
Without trust, the whole system falls apart. It's the trust of the public, an elusive asset once it has been betrayed, that science depends upon for its very existence. Scientific research is overwhelmingly financed by tax dollars, and the need for the goodwill of the public is more than an abstraction.
The Macchiarini affair raises a profound question of trust and responsibility: Should multiple co-authors be held responsible for a lead author's misconduct?
Karolinska apparently believes so. When the institution at last owned up to the scandal, it vindictively found Karl Henrik-Grinnemo, one of the whistleblowers, guilty of scientific misconduct as well. It also designated two other whistleblowers as "blameworthy" for their roles as co-authors of the papers on which Macchiarini was the lead author.
As a result, the whistleblowers' reputations and employment prospects have become collateral damage. Accusations of research misconduct can be a career killer. Research grants dry up, employment opportunities evaporate, publishing becomes next to impossible, and collaborators vanish into thin air.
Grinnemo contends that co-authors should only be responsible for their discrete contributions, not for the data supplied by others.
"Different aspects of a paper are highly specialized," he says, "and that's why you have multiple authors. You cannot go through every single bit of data because you don't understand all the parts of the article."
This is especially true in multidisciplinary, translational research, where there are sometimes 20 or more authors. "You have to trust co-authors, and if you find something wrong you have to notify all co-authors. But you couldn't go through everything or it would take years to publish an article," says Grinnemo.
Though the pressures facing scientists are very real, the problem of misconduct is not inevitable. Along with increased support from governments and industry, a change in academic culture that emphasizes quality over quantity of published studies could help encourage meritorious research.
But beyond that, trust will always play a role when numerous specialists unite to achieve a common goal: the accumulation of knowledge that will promote human health, wealth, and well-being.
[Correction: An earlier version of this story mistakenly credited The New York Times with breaking the news of the Anversa retractions, rather than Retraction Watch and STAT, which jointly published the exclusive on October 14th. The piece in the Times ran on October 15th. We regret the error.]
Masks and Distancing Won't Be Enough to Prevent School Outbreaks, Latest Science Suggests
Never has the prospect of "back to school" seemed so ominous as it does in 2020. As the number of COVID-19 cases climb steadily in nearly every state, the prospect of in-person classes are filling students, parents, and faculty alike with a corresponding sense of dread.
The notion that children are immune or resistant to SARS-CoV-2 is demonstrably untrue.
The decision to resume classes at primary, secondary, and collegiate levels is not one that should be regarded lightly, particularly as coronavirus cases skyrocket across the United States.
What should be a measured, data-driven discussion that weighs risks and benefits has been derailed by political talking points. President Trump has been steadily advocating for an unfettered return to the classroom, often through imperative "OPEN THE SCHOOLS!!!" tweets. In July, Secretary of Education Betsy DeVos threatened to withhold funding from schools that did not reopen for full-time, in-person classes, despite not having the authority to do so. Like so many public health issues, opening schools in the midst of a generational pandemic has been politicized to the point that the question of whether it is safe to do so has been obscured and confounded. However, this question still deserves to be examined based on evidence.
What We Know About Kids and COVID-19
Some arguments for returning to in-person education have focused on the fact that children and young adults are less susceptible to severe disease. In some cases, people have stated that children cannot be infected, pointing to countries that have resumed in-person education with no associated outbreaks. However, those countries had extremely low community transmission and robust testing and surveillance.
The notion that children are immune or resistant to SARS-CoV-2 is demonstrably untrue: children can be infected, they can become sick, and, in rare cases, they can die. Children can also transmit the virus to others, especially if they are in prolonged proximity to them. A Georgia sleepaway camp was the site of at least 260 cases among mostly children and teenagers, some as young as 6 years old. Children have been shown to shed infectious virus in their nasal secretions and have viral loads comparable to adults. Children can unquestionably be infected with SARS-CoV-2 and spread it to others.
The more data emerges, the more it appears that both primary and secondary schools and universities alike are conducive environments for super-spreading. Mitigating these risks depends heavily on individual schools' ability to enforce reduction measures. So far, the evidence demonstrates that in most cases, schools are unable to adequately protect students or staff. A school superintendent from a small district in Arizona recently described an outbreak that occurred among staff prior to in-person classes resuming. Schools that have opened so far have almost immediately reported new clusters of cases among students or staff.
This is because it is impossible to completely eliminate risk even with the most thoughtful mitigation measures when community transmission is high. Risk can be reduced, but the greater the likelihood that someone will be exposed in the community, the greater the risk they might pass the virus to others on campus or in the classroom.
There are still many unknowns about SARS-CoV-2 transmission, but some environments are known risks for virus transmission: enclosed spaces with crowds of people in close proximity over extended durations. Transmission is thought to occur predominantly through inhaled aerosols or droplets containing SARS-CoV-2, which are produced through common school activities such as breathing, speaking, or singing. Masks reduce but do not eliminate the production of these aerosols. Implementing universal mask-wearing and physical distancing guidelines will furthermore be extraordinarily challenging for very young children.
Smaller particle aerosols can remain suspended in the air and accumulate over time. In an enclosed space where people are gathering, such as a classroom, this renders risk mitigation measures such as physical distancing and masks ineffective. Many classrooms at all levels of education are not conducive to improving ventilation through low-cost measures such as opening windows, much less installing costly air filtration systems.
As a risk reduction measure, ventilation greatly depends on factors like window placement, window type, room size, room occupancy, building HVAC systems, and overall airflow. There isn't much hard data on the specific effects of ventilation on virus transmission, and the models that support ventilation rely on assumptions based on scant experimental evidence that doesn't account for virologic parameters.
There is also no data about how effective air filtration or UV systems would be for SARS-CoV-2 transmission risk reduction, so it's hard to say if this would result in a meaningful risk reduction or not. We don't have enough data outside of a hospital setting to support that ventilation and/or filtration would significantly reduce risk, and it's impractical (and most likely impossible in most schools) to implement hospital ventilation systems, which would likely require massive remodeling of existing HVAC infrastructure. In a close contact situation, the risk reduction might be minimal anyway since it's difficult to avoid exposure to respiratory aerosols and droplets a person is exhaling.
You'd need to get very low rates in the local community to open safely in person regardless of other risk reduction measures, and this would need to be complemented by robust testing and contact tracing capacity.
Efforts to resume in-person education depend heavily on school health and safety plans, which often rely on self-reporting of symptoms due to insufficient testing capacity. Self-reporting is notoriously unreliable, and furthermore, SARS-CoV-2 can be readily transmitted by pre-symptomatic individuals who may be unaware that they are sick, making testing an essential component of any such plan. Primary and secondary schools are faced with limited access to testing and no funds to support it. Even in institutions that include a testing component in their reopening plans, this is still too infrequent to support the full student body returning to campus.
Economic Conflicts of Interest
Rebecca Harrison, a PhD candidate at Cornell University serving on the campus reopening committee, is concerned that her institution's plan places too much faith in testing capacity and is over-reliant on untested models. Harrison says that, as a result, students are being implicitly encouraged to return to campus and "very little has been done to actively encourage students who are safe and able to stay home, to actually stay home."
Harrison also is concerned that her institution "presumably hopes to draw students back from the safety of their parents' basements to (re)join the residential campus experience ... and drive revenue." This is a legitimate concern. Some schools may be actively thwarting safety plans in place to protect students based on financial incentives. Student athletes at Colorado State have alleged that football coaches told them not to report COVID-19 symptoms and are manipulating contact tracing reports.
Public primary and secondary schools are not dependent on student athletics for revenue, but nonetheless are susceptible to state and federal policies that tie reopening to budgets. If schools are forced to make decisions based on a balance sheet, rather than the health and safety of students, teachers, and staff, they will implement health and safety plans that are inadequate. Schools will become ground zero for new clusters of cases.
Looking Ahead: When Will Schools Be Able to Open Again?
One crucial measure is the percent positivity rate in the local community, the number of positive tests based on all the tests that are done. Some states, like California, have implemented policies guiding the reopening of schools that depend in part on a local community's percent positivity rate falling under 8 percent, among other benchmarks including the rate of new daily cases. Currently, statewide, test positivity is below 7%, with an average of 3 new daily cases per 1000 people per day. However, the California department of health acknowledges that new cases per day are underreported. There are 6.3 million students in the California public school system, suggesting that at any given time, there could be nearly 20,000 students who might be contagious, without accounting for presymptomatic teachers and staff. In the classroom environment, just one of those positive cases could spread the virus to many people in one day despite masks, distancing, and ventilation.
You'd need to get very low rates in the local community to open safely in person regardless of other risk reduction measures, and this would need to be complemented by robust testing and contact tracing capacity. Only with rapid identification and isolation of new cases, followed by contact tracing and quarantine, can we break chains of transmission and prevent further spread in the school and the larger community.
None of these safety concerns diminish the many harms associated with the sudden and haphazard way remote learning has been implemented. Online education has not been effective in many cases and is difficult to implement equitably. Young children, in particular, are deprived of the essential social and intellectual development they would normally get in a classroom with teachers and their peers. Parents of young children are equally unprepared and unable to provide full-time instruction. Our federal leadership's catastrophic failure to contain the pandemic like other countries has put us in this terrible position, where we must choose between learning or spreading a deadly pathogen.
Blame aside, parents, educators, and administrators must decide whether to resume in-person classes this fall. Those decisions should be based on evidence, not on politics or economics. The data clearly shows that community transmission is out of control throughout most of the country. Thus, we ignore the risk of school outbreaks at our peril.
[Editor's Note: Here's the other essay in the Back to School series: 5 Key Questions to Consider Before Sending Your Child Back to School.]
The Science Sleuth Holding Fraudulent Research Accountable
Introduction by Mary Inman, Whistleblower Attorney
For most people, when they see the word "whistleblower," the image that leaps to mind is a lone individual bravely stepping forward to shine a light on misconduct she has witnessed first-hand. Meryl Streep as Karen Silkwood exposing safety violations observed while working the line at the Kerr-McGee plutonium plant. Matt Damon as Mark Whitacre in The Informant!, capturing on his pocket recorder clandestine meetings between his employer and its competitors to fix the price of lysine. However, a new breed of whistleblower is emerging who isn't at the scene of the crime but instead figures it out after the fact through laborious review of publicly available information and expert analysis. Elisabeth Bik belongs to this new class of whistleblower.
"There's this delicate balance where on one hand we want to spread results really fast as scientists, but on the other hand, we know it's incomplete, it's rushed and it's not great."
Using her expertise as a microbiologist and her trained eye, Bik studies publicly available scientific papers to sniff out potential irregularities in the images that suggest research fraud, later seeking retraction of the offending paper from the journal's publisher. There's no smoking gun, no first-hand account of any kind. Just countless hours spent reviewing scores of scientific papers and Bik's skills and dedication as a science fraud sleuth.
While Bik's story may not as readily lend itself to the big screen, her work is nonetheless equally heroic. By tirelessly combing scientific papers to expose research fraud, Bik is playing a vital role in holding the scientific publishing process accountable and ensuring that misleading information does not spread unchecked. This is important work in any age, but particularly so in the time of COVID, where we can ill afford the setbacks and delays of scientists building on false science. In the present climate, where science is politicized and scientific principles are under attack, strong voices like Bik's must rise above the din to ensure the scientific information we receive, and our governments act upon, is accurate. Our health and wellbeing depend on it.
Whistleblower outsiders like Bik are challenging the traditional concept of what it means to be a whistleblower. Fortunately for us, the whistleblower community is a broad church. As with most ecosystems, we all benefit from a diversity of voices —whistleblower insiders and outsiders alike. What follows is an illuminating conversation between Bik, and Ivan Oransky, the co-founder of Retraction Watch, an influential blog that reports on retractions of scientific papers and related topics. (Conversation facilitated by LeapsMag Editor-in-Chief Kira Peikoff)
Elisabeth Bik and Ivan Oransky.
(Photo credits Michel & Co Photography, San Jose, CA and Elizabeth Solaka)
Ivan
I'd like to hear your thoughts, Elisabeth, on an L.A. Times story, which was picking up a preprint about mutations and the novel coronavirus, alleging that the virus is mutating to become more infectious – even though this conclusion wasn't actually warranted.
Elisabeth
A lot of the news around it is picking up on one particular side of the story that is maybe not that much exaggerated by the scientists. I don't think this paper really showed that the mutations were causing the virus to be more virulent. Some of these viruses continuously mutate and mutate and mutate, and that doesn't necessarily make a strain more virulent. I think in many cases, a lot of people want to read something in a paper that is not actually there.
Ivan
The tone level, everything that's being published now, it's problematic. It's being rushed, here it wasn't even peer-reviewed. But even when they are peer-reviewed, they're being peer-reviewed by people who often aren't really an expert in that particular area.
Elisabeth
That's right.
Ivan
To me, it's all problematic. At the same time, it's all really good that it's all getting out there. I think that five or 10 years ago, or if we weren't in a pandemic, maybe that paper wouldn't have appeared at all. It would have maybe been submitted to a top-ranked journal and not have been accepted, or maybe it would have been improved during peer review and bounced down the ladder a bit to a lower-level journal.
Yet, now, because it's about coronavirus, it's in a major newspaper and, in fact, it's getting critiqued immediately.
Maybe it's too Pollyanna-ish, but I actually think that quick uploading is a good thing. The fear people have about preprint servers is based on this idea that the peer-reviewed literature is perfect. Once it is in a peer-reviewed journal, they think it must have gone through this incredible process. You're laughing because-
Elisabeth
I am laughing.
Ivan
You know it's not true.
Elisabeth
Yes, we both know that. I agree and I think in this particular situation, a pandemic that is unlike something our generation has seen before, there is a great, great need for fast dissemination of science.
If you have new findings, it is great that there is a thing called a preprint server where scientists can quickly share their results, with, of course, the caveat that it's not peer-reviewed yet.
It's unlike the traditional way of publishing papers, which can take months or years. Preprint publishing is a very fast way of spreading your results in a good way so that is what the world needs right now.
On the other hand, of course, there's the caveat that these are brand new results and a good scientist usually thinks about their results to really interpret it well. You have to look at it from all sides and I think with the rushed publication of preprint papers, there is no such thing as carefully thinking about what results might mean.
So there's this delicate balance where on one hand we want to spread results really fast as scientists, but on the other hand, we know it's incomplete, it's rushed and it's not great. This might be hard for the general audience to understand.
Ivan
I still think the benefits of that dissemination are more positive than negative.
Elisabeth
Right. But there's also so many papers that come out now on preprint servers and most of them are not that great, but there are some really good studies in there. It's hard to find those nuggets of really great papers. There's just a lot of papers that come out now.
Ivan
Well, you've made more than a habit of finding problems in papers. These are mostly, of course, until now published papers that you examined, but what is this time like for you? How is it different?
Elisabeth
It's different because in the beginning I looked at several COVID-19-related papers that came out and wrote some critiques about it. I did experience a lot of backlash because of that. So I felt I had to take a break from social media and from writing about COVID-19.
I focused a little bit more on other work because I just felt that a lot of these papers on COVID-19 became so politically divisive that if you tried to be a scientist and think critically about a paper, you were actually assigned to a particular political party or to be against other political parties. It's hard for me to be sucked into the political discussion and to the way that our society now is so completely divided into two camps that seem to be not listening to each other.
Ivan
I was curious about that because I've followed your work for a number of years, as you know, and certainly you have had critics before. I'm thinking of the case in China that you uncovered, the leading figure in the Chinese Academy who was really a powerful political figure in addition to being a scientist.
Elisabeth
So that was a case in which I found a couple of papers at first from a particular group in China, and I was just posting on a website called PubPeer, where you can post comments, concerns about papers. And in this case, these were image duplication issues, which is my specialty.
I did not realize that the group I was looking at at that moment was led by one of the highest ranked scientists in China. If I had known that, I would probably not have posted that under my full name, but under a pseudonym. Since I had already posted, some people were starting to send me direct messages on Twitter like, "OMG, the guy you're posting about now is the top scientist in China so you're going to have a lot of backlash."
Then I decided I'll just continue doing this. I found a total of around 50 papers from this group and posted all of them on PubPeer. That story quickly became a very popular story in China: number two on Sina Weibo, a social media site in China.
I was surprised it wasn't suppressed by the Chinese government, it was actually allowed by journalists that were writing about it, and I didn't experience a lot of backlash because of that.
Actually the Chinese doctor wrote me an email saying that he appreciated my feedback and that he would look into these cases. He sent a very polite email so I sent him back that I appreciated that he would look into these cases and left it there.
Ivan
There are certain subjects that I know when we write about them in Retraction Watch, they have tended in the past to really draw a lot of ire. I'm thinking anything about vaccines and autism, anything about climate change, stem cell research.
For a while that last subject has sort of died down. But now it's become a highly politically charged atmosphere. Do you feel that this pandemic has raised the profile of people such as yourself who we refer to as scientific sleuths, people who look critically and analytically at new research?
Elisabeth
Yeah, some people. But I'm also worried that some people who are great scientists and have shown a lot of critical thinking are being attacked because of that. If you just look at what happened to Dr. Fauci, I think that's a prime example. Where somebody who actually is very knowledgeable and very cautious of new science has not been widely accepted as a great leader, in our country at least. It's sad to see that. I'm just worried how long he will be at his position, to be honest.
Ivan
We noticed a big uptick in our traffic in the last few days to Retraction Watch and it turns out it was because someone we wrote about a number of years ago has really hopped on the bandwagon to try and discredit and even try to have Dr. Fauci fired.
It's one of these reminders that the way people think about scientists has, in many cases, far more to do with their own history or their own perspective going in than with any reality or anything about the science. It's pretty disturbing, but it's not a new thing. This has been happening for a while.
You can go back and read sociologists of science from 50-60 years ago and see the same thing, but I just don't think that it's in the same way that it is now, maybe in part because of social media.
Elisabeth
I've been personally very critical about several studies, but this is the first time I've experienced being attacked by trolls and having some nasty websites written about me. It is very disturbing to read.
"I don't think that something that's been peer-reviewed is perfect and something that hasn't been peer reviewed, you should never bother reading it."
Ivan
It is. Yet you have been a fearless and vocal critic of some very high-profile papers, like the infamous French study about hydroxychloroquine.
Elisabeth
Right, the paper that came out was immediately tweeted by the President of the United States. At first I thought it was great that our President tweeted about science! I thought that was a major breakthrough. I took a look at this paper.
It had just come out that day, I believe. The first thing I noticed is that it was accepted within 24 hours of being submitted to the journal. It was actually published in a journal where one of the authors is the editor-in-chief, which is a huge conflict of interest, but it happens.
But in this particular case, there were also a lot of flaws with the study and that, I think, should have been caught during peer review. The paper was first published on a preprint server and then within 24 hours or so it was published in that paper, supposedly after peer review.
There were very few changes between the preprint version and the peer review paper. There were just a couple of extra lines, extra sentences added here and there, but it wasn't really, I think, critically looked at. Because there were a lot of things that I thought were flaws.
Just to go over a couple of them. This paper showed supposedly that people who were treated with hydroxychloroquine and azithromycin were doing much better by clearing their virus much faster than people who were not treated with these drugs.
But if you look carefully at the paper there were a couple of people who were left out of the study. So they were treated with hydroxychloroquine, but they were not shown in the end results of the paper. All six people who were treated with the drug combination were clearing the virus within six days, but there were a couple of others who were left out of the study. They also started the drug combination, but they stopped taking the drugs for several reasons and three of them were admitted to the intensive care, one died, one had some side effects and one apparently walked out of the hospital.
They were left out of the study but they were actually not doing very well with the drug combination. It's not very good science if you leave out people who don't do very well with your drug combination in your study. That was one of my biggest critiques of the paper.
Ivan
What struck us about that case was, in addition to what you, of course, mentioned, the fact that Trump tweeted it and was talking about hydroxychloroquine, was that it seemed to be a perfect example of, "well, it was in a peer review journal." Yeah, it was a preprint first, but, well, it's a peer review journal. And yet, as you point out, when you look at the history of the paper, it was accepted in 24 hours.
If you talk to most scientists, the actual act of a peer review, once you sit down to do it and can concentrate, a good one takes, again, these are averages, but four hours, a half a day is not unreasonable. So you had to find three people who could suddenly review this paper. As you pointed out, it was in a journal where one of the authors was editor.
Then some strange things also happened, right? The society that actually publishes the journal, they came out with a statement saying this wasn't up to our standards, which is odd. Then Elsevier came in, they're the ones who are actually contracted to publish the journal for the society. They said, basically, "Oh, we're going to look into this now too."
It just makes you wonder what happened before the paper was actually published. All the people who were supposed to have been involved in doing the peer review or checking on it are clearly very distraught about what actually happened. It's that scene from Casablanca, "I'm shocked, shocked there's gambling going on here." And then, "Your winnings, sir."
Elisabeth
Yes.
Ivan
And I don't actually blame the public, I don't blame reporters for getting a bit confused about what it all means and what they should trust. I don't think trust is a binary any more than anything else is a binary. I don't think that something that's been peer-reviewed is perfect and something that hasn't been peer reviewed, you should never bother reading it. I think everything is much more gray.
Yet we've turned things into a binary. Even if you go back before coronavirus, coffee is good for you, coffee is bad for you, red wine, chocolate, all the rest of it. A lot of that is because of this sort of binary construct of the world for journalists, frankly, for scientists that need to get their next grants. And certainly for the general public, they want answers.
On the one hand, if I had to choose what group of experts, or what field of human endeavor would I trust with finding the answer to a pandemic like this, or to any crisis, it would absolutely be scientists. Hands down. This is coming from someone who writes about scientific fraud.
But on the other hand, that means that if scientists aren't clear about what they don't know and about the nuances and about what the scientific method actually allows us to do and learn, that just sets them up for failure. It sets people like Dr. Fauci up for failure.
Elisabeth
Right.
Ivan
It sets up any public health official who has a discussion about models. There's a famous saying: "All models are wrong, but some are useful."
Just because the projections change, it's not proof of wrongness, it's not proof that the model is fatally flawed. In fact, I'd be really concerned if the projections didn't change based on new information. I would love it if this whole episode did lead to a better understanding of the scientific process and how scientific publishing fits into that — and doesn't fit into it.
Elisabeth
Yes, I'm with you. I'm very worried that the general audience's perspective is based on maybe watching too many movies where the scientist comes up with a conclusion one hour into the movie when everything is about to fail. Like that scene in Contagion where somebody injects, I think, eight monkeys, and one of the monkeys survives and boom we have the vaccine. That's not really how science works. Everything takes many, many years and many, many applications where usually your first ideas and your first hypothesis turn out to be completely wrong.
Then you go back to the drawing board, you develop another hypothesis and this is a very reiterative process that usually takes years. Most of the people who watch the movie might have a very wrong idea and wrong expectations about how science works. We're living in the movie Contagion and by September, we'll all be vaccinated and we can go on and live our lives. But that's not what is going to happen. It's going to take much, much longer and we're going to have to change the models every time and change our expectations. Just because we don't know all the numbers and all the facts yet.
Ivan
Generally it takes a fairly long time to change medical practice. A lot of times people see that as a bad thing. What I think that ignores, or at least doesn't take into as much account as I would, is that you don't want doctors and other health care professionals to turn on a dime and suddenly switch. Unless, of course, it turns out there was no evidence for what you were looking at.
It's a complicated situation.
Everybody wants scientists to be engineers, right?
Elisabeth
Right.
Ivan
I'm not saying engineering isn't scientific, nor am I saying that science is just completely whimsical, but there's a different process. It's a different way of looking at things and you can't just throw all the data into a big supercomputer, which is what I think a lot of people seem to want us to do, and then the obvious answer will come out on the other side.
Elisabeth
No. It's true and a lot of engineers suddenly feel their inherent need to solve this as a problem. They're not scientists and it's not building a bridge over a big river. But we're dealing with something that is very hard to solve because we don't understand the problem yet. I think scientists are usually first analyzing the problem and trying to understand what the problem actually is before you can even think about a solution.
Ivan
I think we're still at the understanding the problem phase.
Elisabeth
Exactly. And going back to the French group paper, that promised such a result and that was interpreted as such by a lot of people including presidents, but it's a very rare thing to find a medication that will have a 100% curation rate. That's something that I wish the people would understand better. We all want that to happen, but it's very unlikely and very unprecedented in the best of times.
Ivan
I would second that and also say that the world needs to better value the work that people like Elisabeth and others are doing. Because we're not going to get to a better answer if we're not rigorous about scrutinizing the literature and scrutinizing the methodology and scrutinizing the results.
"I quit my job to be able to do this work."
It's a relatively new phenomenon that you're able to do this at any scale at all, and even now it's at a very small scale. Elisabeth mentioned PubPeer and I'm a big fan — also full disclosure, I'm on their board of directors as a volunteer — it's a very powerful engine for readers and journal editors and other scientists to discuss issues.
And Elisabeth has used it really, really well. I think we need to start giving credit to people like that. And, also creating incentives for that kind of work in a way that science hasn't yet.
Elisabeth
Yeah. I quit my job to be able to do this work. It's really hard to combine it with a job either in academia or industry because we're looking for or criticizing papers and it's hard when you are still employed to do that.
I try to make it about the papers and do it in a polite way, but still it's a very hard job to do if you have a daytime job and a position and a career to worry about. Because if you're critical of other academics, that could actually mean the end of your career and that's sad. They should be more open to polite criticism.
Ivan
And for the general public, if you're reading a newspaper story or something online about a single study and it doesn't mention any other studies that have said the same thing or similar, or frankly, if it doesn't say anything about any studies that contradicted it, that's probably also telling you something.
Say you're looking at a huge painting of a shoreline, a beach, and a forest. Any single study is just a one-centimeter-by-one-centimeter square of any part of that canvas. If you just look at that, you would either think it was a painting of the sea, of a beach, or of the forest. It's actually all three of those things.
We just need to be patient, and that's very challenging to us as human beings, but we need to take the time to look at the whole picture.
DISCLAIMER: Neither Elisabeth Bik nor Ivan Oransky was compensated for participation in The Pandemic Issue. While the magazine's editors suggested broad topics for discussion, consistent with Bik's and Oransky's work, neither they nor the magazine's underwriters had any influence on their conversation.
[Editor's Note: This article was originally published on June 8th, 2020 as part of a standalone magazine called GOOD10: The Pandemic Issue. Produced as a partnership among LeapsMag, The Aspen Institute, and GOOD, the magazine is available for free online.]
Kira Peikoff was the editor-in-chief of Leaps.org from 2017 to 2021. As a journalist, her work has appeared in The New York Times, Newsweek, Nautilus, Popular Mechanics, The New York Academy of Sciences, and other outlets. She is also the author of four suspense novels that explore controversial issues arising from scientific innovation: Living Proof, No Time to Die, Die Again Tomorrow, and Mother Knows Best. Peikoff holds a B.A. in Journalism from New York University and an M.S. in Bioethics from Columbia University. She lives in New Jersey with her husband and two young sons. Follow her on Twitter @KiraPeikoff.