Anecdotes, testimonials and urban legends.

Good storytellers can suck us right in, and for that reason, stories offered as evidence are red flags.

How to recognise this tactic

Those who use this tactic try to present stories about specific cases or events as supporting evidence. The stories range from personal testimonials, to anecdotes about acquaintances, to tales about unidentifiable subjects.

Why do people use this tactic?

People use this tactic when they don’t have proper scientific evidence from controlled tests to support their claims. Most of us find the tactic difficult to resist. In comes from our innate tendency to believe that if two events are correlated, then one must be the cause of the other – the ‘correlation means cause’ fallacy.

What’s wrong with this tactic

Many anecdotes are simply untrue. Anecdotal evidence suffers from two imperfections of human perception and memory (as outlined by Chabris and Simons in The Invisible Gorilla):

• Inattentional blindness: the failure to notice unexpected events when concentrating on a task
• False memory: the distortion of memories to fit in with beliefs, goals and expectations

Those anecdotes that are true are likely to be unrepresentative of the overall situation. In some cases, they are deliberately cherry-picked to be unrepresentative.

Anecdotal evidence works on most people because they fall for the ‘correlation proves causation’ fallacy. If some outcome occurs after we take a certain action, this does not mean that the action caused the outcome. To make a conclusion about a cause-effect relationship, a large number of cases needs to be examined.

Anecdotal evidence is the least reliable type of evidence. If it is used at all, it should only be as an indication that further, properly controlled studies are needed.

What to do when confronted by this tactic

First, ask yourself whether the anecdote is likely to be true. Was it a one-off? What are the motives of the person presenting it? Are there independent witnesses? Could it be the result of incorrect perception or distortion of memory?

If the anecdote could plausibly be true, ask for more evidence. Remember that extraordinary claims require extraordinary evidence. And the plural of anecdotes is not evidence, it’s just more anecdotes. Are there any controlled studies that have examined the phenomenon? (As usual, for medical and health claims, try the Cochrane Library). If so, how does this anecdote fit into the statistics?

Above all, remember that correlation does not prove cause, and don’t act hastily on the basis of any anecdote – seek better evidence.

Variations and related tactics

Inappropriate use of anecdotal evidence is closely related to:

• cherry-picking – selecting a narrow range  of data which seems to support a claim, while ignoring the great majority of evidence that refutes it.
• post hoc ergo propter hoc (after this, therefore because of this) – claiming that because B follows A, then A is the cause of B; the ‘correlation means cause’ fallacy.
• clustering illusion – the tendency to see small ‘clusters’ of data in a random distribution as indicating some significant ‘influence’.

The harmful effects of anecdotal evidence can be magnified by what’s known as an availability cascade – someone comes up with an anecdote which seems to explain a phenomenon and because of its apparent insight, the idea spreads rapidly via social networks, in a chain reaction.

Examples

• A classical case of an anecdote evoking the ‘correlation means cause’ fallacy appears in this testimonial from the website of the Bondi Wheatgrass Company:

I am a young mother of a busy six year old boy. I was beginning to feel very run down and lethargic. After taking wheatgrass juice, my energy levels have risen and I can chase my son all day. My son has also benefited from wheatgrass and has managed to avoid all the usual germs and sicknesses from kindergarten and school.

• A persistent anecdote is used to argue against the wearing of  seatbelts in cars. The story describes how a driver who was not wearing a seatbelt was thrown out of a car in an accident, luckily landed on soft ground and suffered only minor injuries, while the car burst into flames which would have taken his life if he had been strapped in.
  Although this anecdote may be true, it is unrepresentative. Surveys consistently show (here and here) the complete picture – that seatbelts overwhelmingly save lives rather than the opposite.
• Tabloid newspapers are fond of reporting anecdotes that seem to fly in the face of scientific findings. Here’s one that ‘proves’ smoking isn’t bad for you.
• Scientific studies show (here, here and here) that there is no connection between MMR vaccine and autism. Yet some parents continue to refuse to have their children vaccinated on the basis of emotional anecdotes repeated by grieving parents of autistic children. The damage done by the spread of these anecdotes is immense.
• Proponents of “complementary and alternative medicine” (CAM) are fond of using anecdotes in attempting to demonstrate the effectiveness of treatments. Here‘s an article from the Science-Based Medicine blog explaining in detail why anecdotes are not reliable in medical research. And another from Orac at Respectful Insolence.
• William M. London has an interesting account of the way anecdotes can take hold. A TV program reports on a retired medical doctor who is diagnosed with metastatic pancreatic cancer and told he has six weeks to live. He declines chemotherapy and instead embarks on a course of natural and spiritual therapy. The patient and the TV program are full of optimism. London searches for a follow-up story, but all he can find is an obituary, a few months after the program aired.
• Update 2014/03/02: An ironic look at anecdotal evidence:
  

Further material

How Feedback Biases Give Ineffective Medical Treatments a Good Reputation from the Journal of Medical Internet Research.

Why research beats anecdote in our search for knowledge from The Conversation.

What to trust in a “post-truth” world, a TED talk by Alex Edmans, Professor of Finance at the London Business School.

Add to the list of examples by leaving a comment.

The cartoon is courtesy of SMBC.
Michael Shermer’s quote is from Scientific American, November 25, 2008

This is one of ScienceOrNot’s Science red flags. See them all here.