Trusting the experts

Let’s check the science: How can I decide which experts to trust on scientific issues?

Most of us can’t hope to fully understand every scientific issue that impinges on our lives. Even scientists at the pinnacle of their own fields are novices outside those fields. Nearly always, we must turn to the opinions of experts in the field which concerns us. But how do we decide who to trust, who is the real expert? The world is full of pretenders who are eager to persuade us that they have discovered the indisputable truth on issues from cosmic consciousness to the secrets of perfect health. How can you judge who is really an expert?

Quite often, when an idea that could be helpful presents itself, we do not appreciate it, for it is so inconspicuous. The expert has, perhaps, no more ideas than the inexperienced, but appreciates more what he has and uses it better.

 George Pólya, Hungarian mathematician (1957)

What is an expert?

I’ll regard a person as an expert in a field if she:

• has superior factual knowledge in the field and exceptional understanding of how to apply the facts to produce concrete results.
• has a better ability than her peers to differentiate between similar, but not identical cases.
• shows consistency in judgements that is better than her peers.
• is identified by her peers in the field as an expert.

(These criteria are my own and I don’t claim any expertise in the field of experts. They are a kind of condensation of the characteristics of experts used by Ericsson et al and Shanteau et al in references 1 and 2 below.)

Becoming an expert is not easy. Ericsson et al¹ explain:

The journey to truly superior performance is neither for the faint of heart nor for the impatient. The development of genuine expertise requires struggle, sacrifice, and honest, often painful self-assessment. There are no shortcuts. It will take you at least a decade to achieve expertise, and you will need to invest that time wisely, by engaging in “deliberate” practice— practice that focuses on tasks beyond your current level of competence and comfort.

Deciding which experts to trust

Alvin I. Goldman (see reference 3 below) has proposed five criteria for deciding which experts to trust. His criteria are meant to be used to select which of two competing experts we should prefer. I’ve used his criteria as a basis, but modified and broadened them in an attempt to produce a sort of checklist for deciding whether a particular expert is on the level. (Goldman would probably not approve.)

For each criterion, I’ve given some descriptors that you might be able to use as indicators. Don’t expect any particular expert to tick all the boxes, but obviously, the more the better. I realize that, for many of the criteria, you won’t be able to say one way or the other. Hopefully, though, you’ll find enough to either reject or accept your “expert” candidate without too much trouble.

Red flags when seeking experts

There are some signals that tell you immediately you are not dealing with an expert. You should be very alert to these signals. Among them are:

• Fudging the boundaries of expertise – the putative expert tries to imply that because he’s an expert in one field, his opinion in a different field should be given credence.
• Appeals to testimonials – the putative expert rolls out glowing recommendations from clients or grateful followers.
• The Galileo Gambit – the putative expert insists that he is an unacknowledged genius, a maverick who is shunned by mainstream science because of his unconventional ideas.
• Conspiracy theories – the putative expert claims that some form of “scientific establishment” is suppressing his ideas.
• The Gish Gallop – the putative expert slickly rattles off a long list of assertions without providing evidence or allowing questions.
• Overconfidence – it’s been found that most of us judge the quality of experts by the amount of confidence they project. Unfortunately, the well-known Dunning-Kruger effect means that those who are least competent in a field are precisely the ones who overate their own performance. Those who are most competent are more likely to underrate their performance. True experts will admit that they are never completely certain of their stance and can specify the uncertainties in their explanations.

For an extended list of similar signals, see my Science Red Flags page.

Further reading

Why we need to listen to the real experts in science at The Conversation.

The science of why you really should listen to science and experts by Chris Mooney at The Washington Post.

Whose word should you respect in any debate on science? by Peter Ellerton at The Conversation.

George Pólya’s quote is from How to Solve it: A New Aspect of Mathematical Method (1957), 223.
The Impostor cartoon is from xkcd.
This page is based on ideas developed in these articles:
1. K. Anders Ericsson et al (2007), The Making of an Expert, Harvard Business Review, July – August 2007
2. James Shanteau et al (2003), How Can You Tell if Someone is an Expert? Empirical Assessment of Expertise, in Emerging Perspectives on Judgment and Decision Research, Cambridge, U.K.: Cambridge University Press.
3. Alvin I. Goldman (2001), Experts: Which Ones Should You Trust? Philosophy and Phenomenological Research, Vol. 63, No. 1., pp. 85-110.

This is one of ScienceOrNot’s Let’s check the science series.