Seven years ago, a deeply controversial paper claimed that states introducing laws to expand naloxone led to increases in opioid-related emergency room visits and arrests, in a reflection of “moral hazard.”
The paper, which sparked headlines everywhere from the Atlantic to the Washington Post and CNN, was heavily criticized by harm reduction advocates and other public health experts at the time. A researcher in Australia has now sought to replicate the study—and found major flaws which made its conclusions unjustifiable.
The original study—“The Effects of Naloxone Access Laws on Opioid Abuse, Mortality, and Crime”— came out as a working paper in 2018, with the final version published in the Journal of Law and Economics in 2022. It explored whether laws expanding access to naloxone, the opioid-overdose antidote, might unintentionally increase certain risks.
“We find that broadened access led to more opioid-related emergency room visits and more opioid-related theft, with no net measurable reduction in opioid-related mortality,” wrote its authors, Drs. Jennifer Doleac and Anita Mukherjee. “We conclude that naloxone has a clear and important role in harm-reduction, yet its ability to combat the opioid epidemic’s death toll may be limited without complementary efforts.”
Those findings have been the subject of intense scrutiny ever since. So Dr. Sergey Alexeev, a senior research fellow and health economist at the University of Sydney, decided to see if he could reproduce them using the same methods.
His replication study, published by Economic Inquiry on March 9, found multiple issues with the original methodology, rendering the conclusions “invalid.”
Alexeev noted that the moral hazard debate, when applied to harm reduction tools like naloxone, is a question with critical policy stakes.
The original paper, he wrote in his own study, suggested “that such interventions may inadvertently encourage riskier behavior through the mechanism of moral hazard.”
Alexeev told Filter that pinning moral hazard on naloxone requires hard evidence, not just theory.
“Drug policies are uniquely sensitive to public opinion, underscoring the pressing need for quality and responsible research,” Alexeev wrote. “Doleac and Mukherjee (2022) conclude that laws designed to increase access to Narcan do not reduce mortality and instead increase opioid use intensity. Despite contradicting studies published both before and after, their paper has received widespread publicity in the media and policy think tanks.”
Moral hazard is a concept from economics, originally used to describe insurance. The theory is, if you are protected against something bad happening, you will be less likely to stop behaving in a risky way. For instance, if you have homeowner’s insurance, you might decide against spending money to fix a leak in your house, because you think the insurance company will cover any serious damage. Or, because your car has a seatbelt, you might feel safer driving at higher speeds.
But Alexeev told Filter that pinning moral hazard on naloxone requires hard evidence, not just theory.
“It’s an empirical puzzle,” he said. “Rational users might take bigger risks if overdoses feel less fatal, but addiction’s mix of planning and impulse muddies that. Prior studies haven’t consistently shown this, and my replication found the data doesn’t either.”
The original study tested the theory using three data points: Google search trends for naloxone, opioid-related arrests and opioid-related ER visits.
The authors “interpret these empirical findings as evidence that naloxone makes opioid use safer, leading to riskier drug use, ultimately resulting in a net effect of zero on mortality,” Alexeev wrote.
Doleac and Mukherjee claimed that an increase in Google searches for naloxone showed growing interest for obtaining the medication in states that made it more accessible—but that a decrease in searches for rehab suggested people were less interested in treatment.
In Alexeev’s analysis, however, he was unable to replicate the reported Google search trends in specific cities and states, meaning there was no way to reach such conclusions. “The availability of both high-frequency and spatially granular Google search data is non-existent for unique topics such as naloxone,” he wrote.
Doleac and Mukherjeee meanwhile relied on the United States’ National Incident Based Reporting System (NIBRS) to track opioid-related arrests, finding that these increased alongside legal expansions to naloxone access. But Alexeev was also unable to reproduce their findings here, despite accessing the publicly available data.
“All three datasets on which they based their conclusions, you can’t work with it—it either doesn’t exist or it’s so badly mismanaged. Any of these things, if you fix it, the result does not hold.”
For example, he aggregated NIBRS data for “all thefts” and showed variations over time, with changes by the season of the year—following a downward trend. But when he looked at records for opioid-related thefts, it showed nearly zero for the year 2015, which doesn’t match the original claims.
Alexeev pointed out that the NIBRS is a problematic source of data to begin with, when many law enforcement agencies under-report certain incidents or choose not to order costly drug tests. In general, reported drug-related law violations are not a reliable indicator of actual drug use, because police are only reporting whatever they happen to discover.
Finally, Alexeev looked at the hospitalization data. Here, too, he found discrepancies. Fifteen states had data missing for certain quarters, and 10 states showed data that didn’t match up with the original study. And in a potentially misleading manner, states severely impacted by overdose, like West Virginia and Pennsylvania, were not included in the original study’s hospitalization data but were still referenced in the study.
“I use a stronger design, correcting for the issues,” he told Filter. “Then you get a very precise zero—no effect at all.” Introducing naloxone access laws “does not increase or reduce the opioid ER admissions. There’s no sign of moral hazards.”
Alexeev also found discrepancies in terms of the dates when states enacted naloxone access laws.
“All three datasets on which they based their conclusions, you can’t work with it—it either doesn’t exist or it’s so badly mismanaged,” Alexeev said. “Any of these things, if you fix it, the result does not hold.”
Photograph courtesy of Remedy Alliance
Show Comments