‘The U.S. effect': Researchers encouraged to report exaggerated or eye-catching results
Scientists who study human behaviour are more likely than average to report exaggerated or eye-catching results if they are based in the United States, according to an analysis of more than 1,000 research papers in psychiatry and genetics. This bias could be due to the research culture in the US, authors of the analysis said, which tends to preferentially reward scientists for the novelty and immediate impact of a piece of work over the quality or its long-term contribution to the field.
Daniele Fanelli, University of Edinburgh, one of the authors of the latest analysis, said that there was intense competition in the US for research funds and, subsequently, pressure to report novel findings in prestigious, high-impact scientific journals.
“We don’t know what causes the US effect but we think the most likely explanation is that it’s about the research environment in the US,” he says. “Somehow the researchers there are subtly more pressured than elsewhere in the world to make strong discoveries. This very idea that you do science to make strong discoveries is natural but it’s a problem to science itself. Science should be about doing good, precise studies. Not necessarily about getting exciting new results every time.”
Working in a research environment where careers depend on publishing the most exciting and strongest results might unconsciously draw researchers to exaggerate their findings, he added. “The problem, if we’re right about the US effect, is that the US itself should re-think the way they are rewarding researchers. They shouldn’t reward researchers only because they get a lot of papers in a lot of high-ranking journals. They should reward research that is methodologically highly accurate.”
Fanelli worked with John Ioannidis of Stanford University on the study. They looked at 1,172 results reported on in 82 meta-analyses on a range of biological and behavioural questions in the fields of genetics and psychiatry. Each meta-analysis took the results of 10 to 20 individual research studies that examined the same question, such as whether a particular drug had an effect on a patient in the expected way. For each study, Fabelli and Ionnidis recorded how far the strength of the reported effect was, from the average across many studies, for that question.
They found that behavoural studies were more likely to report more extreme results than studies that involved biological or chemical measurements in a person. Behavioural studies with lead scientists in the US were also more likely than average to report extreme effects that confirmed the initial hypotheses of the studies. The results are published today in the Proceedings of the National Academy of Sciences.
The “US effect”, as the authors called it, is not huge. When studying the strength of a particular effect – the efficacy of a drug, say, or a type of psychiatric intervention – studies with US-based first authors tended to have a 5% stronger effect than the average across all countries. “It’s not that other countries don’t have these kinds of biases,” Fabelli said. “But, compared to what happens on average, the US is a bit above the average.”
Chris Chambers, a psychologist at Cardiff University who did not take part in the study, said bias in psychology research was a problem. “There’s already a lot of evidence showing how common various ‘grey area’ research practices are in the behavioural sciences – these are things like reinventing hypotheses to fit unexpected results and exploiting the lack of standardisation in research practices to produce desirable outcomes.”
The problem with all of these practices, Chambers said, was that they promoted false discoveries and hindered science in a way that did not necessarily self-correct over time. “My belief is that US scientists aren’t actively engaging in mass fraud – instead, most of these questionable practices are unconscious,” he said. “It’s easy to fool ourselves into thinking that a result which ‘feels’ right is in fact true. This problem is known as confirmation bias, which ironically was itself discovered by psychologists.”
Fanelli said his primary concern was not to cause concerns about the quality of US research, nor to accuse scientists there of undue bias. “The problem is more complex than that – the very way we manage science makes a difference on the quality and accuracy of the science itself.”
He said the effect may not be limited to the US for very long. “If we’re right about the causes of the US effect, then what is happening in many countries around the world, definitely in Europe and the UK, you are having similar performance-based systems becoming more important in deciding about scientific careers and grants. Yes, we should worry. We think it is the case that, even if now some countries don’t have their own effect, they are likely to show it more and more in the future.”
Chambers said that many psychologists were already working hard on solutions. “One of them is study pre-registration. We can eliminate most of the problems in behavioural research by asking researchers to register their experimental methods and analyses before they run their experiments. This eliminates much of the wiggle room that allows problems like those highlighted here to arise in the first place.”
He added that it fell to funders of research to take the initiative in leading reform. “Under the current incentive structure in academia, scientists who put the collective good ahead of their own personal career are severely punished. We need funders and regulators to take ownership of this problem and mandate gold-standard working practices to eliminate it. I hope the National Institutes of Health and other US funding agencies read this latest research carefully.”
[Scientist studies molecule via Shutterstock]