Single study syndrome – clutching at convenient confirmation

The notion that the latest scientific information trumps all is a red flag.

How to recognise this tactic

This tactic shows up when a person who has a vested interest in a particular point of view pounces on some new finding which seems to either support or threaten that point of view. It’s usually used in a context where the weight of evidence is against the perpetrator’s view.

… “single-study syndrome” … tends to turn up whenever a political agenda is threatened or supported by a specific line of scientific inquiry. “Some new finding, however tentative, gets highlighted while the broader suite of research on a tough subject is downplayed or ignored, …”

Mark Follman, quoting  Andrew Revkin, American science writer, 2011.

Why do people use this tactic?

People use this tactic to try to convince you that the evidence supports their position. They hope you are fooled into believing that, because a single new study is hot off the presses, its evidence supersedes all previous evidence.

What’s wrong with this tactic

In science, it’s the overall weight of evidence that matters. Unless a new study turns up evidence that falsifies most of the existing evidence, it can’t be considered any more influential than other studies. In most cases, it should be regarded as simply another piece in the jigsaw of evidence, especially while it is still subject to post-publication peer review and possible attempts at replication. It’s not uncommon for faults to be found in recent studies that render the results worthless.

What to do when confronted by this tactic

Don’t let yourself be talked into accepting the latest study as the final word on a topic, even if it supports your own viewpoint. Ask (or find out) how the study fits in with the full spectrum of evidence on the topic. You may be able to find systematic reviews on the topic to help with this. To get some idea of how influential a new study will be, try to tap into the informal post-publication reviews that invariably appear on respectable blogs. Remember that scientific models are tentative and withhold judgement until the quality of the evidence is established.

Variations and related tactics

This tactic derives from confirmation bias, the human tendency to favour information that confirms existing beliefs. It has similar characteristics to the anchoring effect, but in anchoring, the single study is responsible for establishing the initial bias.

Examples

• A recent animal study on the health effects of GM maize was immediately jumped on as the last word on the topic by opponents of GM food (for example, see here) because it claimed to show that the maize produced cancers. Almost immediately, the validity of the study was questioned (here, here, here, here, here), as was the unusual behaviour of the authors in publicising the study (here). The study has now been widely discredited. Animal studies have shown that there are few health concerns associated with GM foods, although one recent review explained that much more investigation is needed.
• In August 2011, climate-deniers seized on a study from CERN that in their eyes proved that climate change is driven by cosmic rays. Multiple independent lines of evidence show, of course, that the earth’s climate is changing as a result of human-produced greenhouse gases. Although the study shed some light on the role of cosmic rays in aerosol formation, in the words of its lead author, “it actually says nothing about a possible cosmic-ray effect on clouds and climate, but it’s a very important first step.” As Skeptical Science explains:

In reality, the CERN experiment only tests the bolded step in this list of requirements for cosmic rays to be causing global warming:

1. Solar magnetic field must be getting stronger
2. The number of cosmic rays reaching Earth must be dropping
3. Cosmic rays must successfully seed clouds, which requires:
   1. Cosmic rays must trigger aerosol (liquid droplet) formation
   2. These newly-formed aerosols must grow sufficiently through condensation to form cloud-condensation nuclei (CCN)
   3. The CCN must lead to increased cloud formation
4. Cloud cover on Earth must be declining

Update 2013/11/12: This article by Dana Nuccitelli explains how a barrage of new papers now show that variations in cosmic rays are not responsible for current global warming. Here‘s another by Rasmus E Benestad at RealClimate.

• In September 2012, the Annals of Internal Medicine published a Stanford University systematic review into the safety and nutritional value of organic foods. It found that:

The published literature lacks strong evidence that organic foods are significantly more nutritious than conventional foods. Consumption of organic foods may reduce exposure to pesticide residues and antibiotic-resistant bacteria.

Another systematic review published 2 years earlier had likewise found no difference in nutritional value between organic and conventional foods. Nevertheless, the Stanford study provoked criticism from supporters of organic food, to the extent that a petition calling for retraction was organised. Even if the paper were retracted (and there are no reasons why it would be), this would have little effect because the papers that were analysed in the study still convey the same message.

• Update 2012/11/21: Here’s an interesting discussion at Climate Progress about a paper called “Little change in global drought over the past 60 years” which appeared in Nature recently. It clashes with the existing literature on the topic and drew immediate criticism from other climate scientists.
• Update 2013/04/15: A recent study (here) gained a huge amount of publicity because it claimed to show once and for all that the Mediterranean diet prolonged life. However, major flaws have been pointed out in this study (here, here).

Further reading

This is why you shouldn’t believe that exciting new medical study by Julia Belluz at Vox.

‘Survival of the wrongest’ by David H. Freedman at the Columbia Journalism Review.

Do we need more studies on vaccines, GMOs, climate change, etc.? by Fallacy Man at The Logic of Science.

---

Mark Follman’s quote is from an article on Mother Jones.
The Neutrinos cartoon is from xkcd

This is one of ScienceOrNot’s Science red flags. See them all here.