Devious deception in displaying data: Cherry picking

We expect scientists to tell the truth: when someone selects only a small part of the whole truth, that’s a red flag

How to recognise this tactic

In cherry-picking, people use legitimate evidence, but not all of the evidence. They select segments of evidence that appear to support their argument and hide or ignore the rest of the evidence which tends to refute it.

Choosing to make selective choices among competing evidence, so as to emphasize those results that support a given position, while ignoring or dismissing any findings that do not support it, is a practice known as “cherry picking” and is a hallmark of poor science or pseudo-science.

Richard Somerville, American climate scientist, 2011

Why do people use this tactic?

People use this tactic when the bulk of evidence goes against them, but they refuse to accept the logical conclusion because they can find periods or occasions or excerpts that seem to support their position. The tactic is commonly found when statistical data are involved, because such data usually show natural variation even though there is an overall trend in one direction.

What’s wrong with this tactic

Cherry-picking is dishonest. Anyone who uses this tactic is misleading you. Admittedly, some people who are non-specialists may not be familiar with all of the evidence in a field, so they don’t knowingly mislead. But more often cherry picking is deliberate. The most charitable you can be about cherry picking is to assume that the perpetrator is suffering from confirmation bias – the unconscious tendency to notice only evidence that supports their idea and ignore less favourable evidence.

What to do when confronted by this tactic

Cherry picking is not easy to spot because perpetrators deliberately avoid any mention of evidence outside the range they present to you. Always maintain a healthy scepticism about evidence, particular if it seems to be restricted in extent (e.g. confined to a narrow time-range, geographical area, sub-population, age range, text extract etc). If this is the case, try to make queries about evidence from outside these confines.

If it turns out that you have been fed cherry-picked data, try to find out the reasons. Does the perpetrator have an agenda or a vested interest in twisting the evidence?

Variations and related tactics

Quoting out of context , or quote mining, is a type of cherry picking. In this tactic, a spoken or written passage is lifted out of its surrounding text in a misleading way.

Use of anecdotal evidence is a form of cherry picking. Anecdotes are usually unrepresentative of the overall situation.

Closely related to cherry picking are attentional bias (noticing only that evidence that seems relevant), confirmation bias (noticing only that evidence that confirms existing beliefs) and anchoring bias (giving undue importance to one piece of evidence and ignoring other evidence).

Examples

• Creationists like to quote mine statements by scientists in attempts to show that there are weaknesses in the evolutionary model. Here are some examples from evolutionary biologist Richard Dawkins.
  
• Global warming deniers frequently claim that global warming has stopped in recent years. In this claim, they are cherry-picking. This graphic from Skeptical Science illustrates how the cherry-pick distorts the situation.
  

  How climate skeptics cherry pick temperature records

Update 2013/03/06: Tamino at Open Mind dissects a perfect example of climate-denier cherry-picking (here). Trenberth and Fasullo explain here why the global surface temperature trend seems to have slowed even though the earth is still warming.

• The stolen  ‘climategate’ emails attained undue prominence in the media because they were quote mined by global warming deniers. The scientists who wrote the emails have been cleared of misconduct by multiple independent inquiries. Peter Hadfield tells the story:
  
• In a classic example of anchoring bias, anti-vaccination campaigners built an edifice of misinformation based on a single 1998 study by Dr. Andrew Wakefield and others, which suggested that MMR vaccine could have caused intestinal disorders in autistic children. Despite the lack of evidence from other studies to support this link, the anti-vaccination movement clung to Wakefield’s findings. The paper was retracted in 2010, and Wakefield has been struck off the Medical Register for serious professional misconduct..
• In 2010, the British Homeopathic Association submitted a memorandum to a UK parliamentary investigation. Numerous scientists later reported that the document had misled through misrepresentation and cherry picking of their work.
• George Monbiot, in the Guardian, describes how the anti-nuclear campaigner Helen Caldicott uses cherry picking and anecdotal evidence, among other tactics, to mislead.

---

Richard Somerville’s quote is from his Testimony before the U.S House of Representatives Committee on Energy and Commerce Subcommittee on Energy and Power, March 8, 2011.

The Escalator temperature graphic is from Skeptical Science Graphics (Skeptical Science) / CC BY 3.0

This is one of ScienceOrNot’s Science red flags. See them all here.