For years medical researchers held off while scientists in other fields embraced online platforms for posting rough drafts of manuscripts, known as preprints.
Those websites accelerate basic science by allowing researchers to disseminate findings and get feedback on their work before submitting them to a traditional journal.
Some argue that such rapid data sharing is ideally suited for infectious disease outbreaks like the one we’re experiencing now.
However, the prospect of public access to unvetted work sparked worry about potential health scares and patients demanding unproven treatments. A BMJ editorial put it this way: “Can the need for speed be balanced with suitable safeguards to protect the public?”
We’re now finding out.
A medical preprint server called medRxiv (pronounced “med-archive”) went live last summer. It’s a partnership of BMJ (publisher of The BMJ), Yale University, and Cold Spring Harbor Laboratory.
MedRxiv has been flooded with COVID-19 research. As of Tuesday, the server hosted 799 preprints about the virus. A related site for biosciences, bioRxiv, had 237.
Predictably, those draft manuscripts have become fodder for news stories as journalists race to report any and all information about the virus.
“Until COVID-19 there was no reason for the press to pay attention (to preprints) because they could count on the journals to curate what they felt was most important, most impactful,” said one of medRxiv’s founders, Joseph Ross, MD, a professor of medicine and public health at the Yale School of Medicine. “But now it’s chaotic. Science is moving really quickly and we’re all trying our best to understand things.”
Unfortunately, some news stories have cited preprints with little or no explanation of what they are and with no strong cautions about their uncertain quality. Unlike published journal articles, preprints haven’t been peer-reviewed by independent experts who might point out errors or weaknesses.
For example, Wired and Vice covered a preprint of a lab study that examined how long the virus lasts on different types of surfaces and in the air. The stories ran after a researcher Tweeted a link:
Both news stories said the data came from a preprint but did not define what a preprint is. Both of these publications target mainstream audiences that can’t be expected to be know what preprints are and how they fit into the research process. (Wired did caution that this was lab data that might not pan out in the real world, but Vice provided no such caveat.)
The medRxiv website was not much help, either. Days later the site stated that the preprint had been “published in The New England Journal of Medicine,” which might suggest that it had been peer-reviewed and published as an article. In fact, it ran as a letter to the editor.
More problematic were numerous sensational news stories based on a preprint claiming people with blood type A have “significantly higher risk for acquiring Covid-19”. Here are a few:
New York Post: People with blood type A might be more susceptible to coronavirus, study finds
Daily Mail: People with Type A blood are MORE likely to catch coronavirus than those with Type O, study claims
WVLT Knoxville: Study says your blood type could make you more susceptible to coronavirus.
The coverage wasn’t void of cautions. The Post story called the study “preliminary” in the lead, and the Daily Mail story mentioned in the fifth paragraph that the research “has yet to be scrutinized by other academics in peer review.” The WVLT story stated at the end: “The study has not yet been peer-reviewed.”
But even if those caveats registered with readers, was this story — based on a single unvetted study — worth reporting at all? Even if proven true — which is far from certain — what difference would it make in testing and treatment?
In days that followed, other news organizations ran stories highlighting the study’s flaws. Some of WVLT’s fellow affiliates in the Cox Media Group ran “fact check” stories that offered more caveats, like one from WPXI Pittsburgh. Medical News Today quoted a researcher who said the results “might be purely coincidental. Importantly, people should not panic about these results, as, clearly, further scientific research is required to substantiate these claims.”
A ‘big red flag’
Steven Woloshin, MD, co-director of the Center for Medicine and Media at The Dartmouth Institute, said journalists should view preprints as “a big red flag” about the quality of evidence, similar to an animal study that doesn’t apply to humans or a clinical trial that lacks a control group.
“I’m not saying the public doesn’t have the right to know this stuff,” Woloshin said. “But these things are by definition preliminary. The bar should be really high” for reporting them.
In some cases, preprints have shown to be completely bogus. As FactCheck.org reported, a preprint suggesting that the new coronavirus might have been created in a lab was shared widely on Twitter and posted on conspiracy websites before it was taken down by bioRxiv.
Readers might not heed caveats about “early” or preliminary” evidence, Woloshin said. “The problem is, once it gets out into the public it’s dangerous because people will assume it’s true or reliable, and I don’t think that’s true.”
Woloshin suggested news organizations refrain from saying a preprint was “published,” which wrongly signals that a manuscript “must have gone through some sort of editorial review.”
Both the Wired story and a New York Times piece about potential drug treatments used the word “published.” By contrast, a STAT story assessing evidence on how long the virus can remain airborne reported that a preprint was “posted to” a preprint site. (The STAT story also took care to explain that the preprint had not been peer-reviewed.)
All data should be ‘scrutinized beyond belief’
Ivan Oransky, MD, vice president of editorial at Medscape and a Distinguished Writer in Residence at New York University’s Arthur Carter Journalism Institute, agreed that the public should be alerted about what preprints are. However, he said journalists should vet all COVID-19 research with equal vigor, whether it’s a preprint or a peer-reviewed journal article.
Oransky is a co-founder of the science blog Retraction Watch, which spotlights problematic journal articles.
“To me the rules for reporting on preprints shouldn’t be any different from reporting on journal articles,” Oransky said. “I think everything needs to be scrutinized beyond belief.”
Arguably the most damaging COVID-19 hype so far sprung from a study that ran in a journal. The malaria medicine hydroxychloroquine, touted by President Trump as a potential “cure,” gained traction based in part on an shaky study of just 42 patients in France.
The study’s authors concluded that the drug, when used in combination with an antibiotic, decreased patients’ levels of the virus. However, the findings were deemed unreliable due to numerous methodological flaws. Patients were not randomized, and six who received the treatment were inappropriately dropped from the study.
The findings made a news splash and led to drug hoarding and deaths as people self-medicated.
The paper appeared to have escaped editorial scrutiny, and it’s been noted that one of the co-authors is the editor-in-chief of the journal that published it. One researcher complained on Twitter:
A call for extra diligence
Despite uneven editorial rigor, Woloshin said journal publication is “still an extra layer of scrutiny that just isn’t there with a preprint.” In sound peer review, analyses are questioned, assertions toned down, and hypotheses challenged, he said. “If they are good reviewers and editors, they make sure the researchers are up front about the limitations and caveats.”
Ross agreed journalists should use extra caution with preprints.
“Preprints themselves are not directed toward the public and the media,” Ross said. “I think a journalist could argue we’re in the middle of a pandemic — that’s a good reason to write about preliminary work. But then they should do extra due diligence.”
That said, medRxiv has “tried to be careful and not just let anything go live,” said Ross.
According to its guidelines, submissions “undergo a basic screening process for offensive and/or non-scientific content and for material that might pose a health risk.” Editorials and case studies are not allowed. Submissions must be registered on a public database of clinical trials and comply with established transparency guidelines.
In addition, warnings on medRxiv and bioRxiv state that preprints “should not be reported in news media as established information.”
While we’ve criticized the promotion of unvetted study abstracts at scientific meetings, Ross believes preprints offer more transparency than those preliminary summaries because they are complete manuscripts. “You can really scrutinize the work in the way you can’t when it presented as an abstract at a conference,” he said.
But it’s still up to journalists to do that scrutiny, including reaching out to independent experts and checking to see whether other research concurs. Those steps aren’t always taken.
Institutions are touting their preprints
Editors and producers also might consider whether the public would truly benefit from knowing about a preprint now, before the scientific community has had a chance to kick the tires. One preprint that falls short of the mark is a preprint covered in Wired about the possible future use of gene-editing technology to fight viruses.
Strong editorial filters are needed as researchers and their institutions tout unvetted work.
Among those that have issued news releases about COVID-19 preprints are the University of Michigan and MIT. Both included prominent caveats about the preliminary nature of the findings.
In an email, Rick Fitzgerald, Michigan’s Assistant Vice President for Public Affairs, provided a rationale for publicizing a preprint about an analysis of containment measures:
In this rapidly changing environment, we can be of greater service to society by getting the word out now, rather than always waiting for traditional peer review. We considered what potential harm could come if the results turned out to be erroneous and we believe the benefits of publicizing were enough to merit an earlier news release in this instance. The paper provides evidence that containment measures work, and if we were to wait six months for it to be peer-reviewed, we could lose opportunities for it to inform how we handle this current pandemic.
It’s anybody’s guess whether news organizations will continue to cover preprints after the current crisis abates. But for now, even the experts can’t make sense of them all.
“You can’t ignore them,” Anthony Fauci, MD, head of the U.S. National Institute of Allergy and Infectious Diseases, told Science magazine. But sometimes he said, “It gets a little confusing what you can really believe.”
