Scientific models are tested by making predictions and checking them against real world data.

Good scientists make models and then try to break them.

This is one of ScienceOrNot’s Hallmarks of science. See them all here.

In short…

Real-world testing of models guards against wishful thinking. A scientific model is accepted as valid only after it is tested against data from the real world and the evidence supports it. Scientists have most confidence in models that are supported by multiple independent lines of evidence.

Scientists are building explanatory structures, telling stories which are scrupulously tested to see if they are stories about real life.

Peter Medawar,  Brazilian-British biologist and medical researcher, 1982

How models are tested.

In science, no model is accepted until it is tested against the real world. Scientists use models to make predictions and then carry out critical tests to check whether those predictions were accurate.

Exactly what should be tested and what the results should be depends on the characteristics of the particular model. Each model should specify what physical circumstances are required and predict what data should be found as a result.  If possible, scientists should choose the testing circumstances so that agreement would be very unlikely if the model is wrong. They then gather real-world data under those circumstances, to see whether the data agree with the predictions.

There are different ways of gathering evidence, depending on the type of model being tested. Scientists use controlled experiments, clinical trials, observation, surveys ….any method that will give accurate, empirical data. (See Science uses controlled experiments, Science uses observation and Science uses randomized controlled trials for more.)

No one expects exact agreement between predicted and measured data: some degree of accuracy or statistical level of significance (e.g. a margin of less than 5%) is always specified (see Science quantifies the uncertainty in its data and conclusions).

If the data do agree, the model is supported but is not necessarily true. Any alternative models should be examined to find whether they give better agreement. If they don’t, the chosen model is probably a good one.

If the predicted and measured data do not agree, the model may be rejected or modified. A modified model must be re-tested using new predictions and new data.

The gold standard of testing is ‘multiple independent lines of evidence‘, meaning that the model has been thoroughly tested and supported by many different investigations. When this happens, we can have great confidence in the model.

Sometimes, there is a lot of evidence in support of a model, and some other evidence that fails to support it. As long as there is no falsifying evidence, scientists would say that the weight of evidence supports the model.

Why testing is needed

Humans are susceptible to self-deception and wishful thinking. We are all tempted to believe that the world is the way we would like it to be. We tend to ignore evidence that would show us it’s not. We clutch at irrelevant, chance events and use them to confirm our beliefs. We build mental models and then twist the ‘facts’ to fit. Testing of models against the real world using predictions guards against these shortcomings.

Bogus science tends to adopt models, hypotheses and theories that have not been tested against the real world. It often appeals to authority, testimonials, ancient wisdom, anecdotal evidence or pseudo-scientific jargon to justify its models. In science, none of these carry any weight. On the rare occasion that bogus science submits to testing, it often relies on a single isolated study.

Examples

• In the eighteenth century, Edward Jenner, an English country doctor, was intrigued by the local belief that people who had contracted the mild disease cowpox were immune to the serious disease smallpox. In 1796, he decided to test the belief by inoculating James Phipps, an 8-year old boy, with material from the cowpox scabs of a dairymaid. A few weeks later, he inoculated the boy again with smallpox microbes. The boy did not contract smallpox, showing that he was immune.
• The Plate Tectonic model of the Earth is supported by multiple independent lines of evidence – magnetic stripes in rocks showing sea-floor spreading, the global distribution of earthquakes and volcanoes, comparable fossils found on widely separated continents, satellite measurements.
• The Anthropogenic Global Warming model proposes that the current global warming and associated climate change are caused principally by human emissions of greenhouse gases. This is supported by multiple independent lines of evidence. For more details, see here.
• Mathematical climate models are tested exhaustively by comparing their predictions with real-world conditions. When they fail in some details, they are modified. Here‘s an explanation of how the process works.
• Astrology attempts to account for human characteristics and predict events by using a model based on the positions of constellations and planets. Real-world tests show that the model fails (see here, here, and here).
• The homeopathy model proposes that an ailment can be treated using highly-diluted solutions of substances that cause symptoms similar to those of the ailment. This model fails real-worlds tests. (See here, and here.)
• Iridology, the notion that the appearance of the iris can be used to make diagnoses about a person’s health, fails real world tests (see here and here).

Peter Medawar’s quote is from Pluto’s Republic, OUP, 1984, p 133. 
The Light interference image is by Andrew Riley-Watson on flickr. Interference patterns such as this one are real-world evidence for the wave model of light.

This is one of ScienceOrNot’s Hallmarks of science. See them all here.

Updated: 2013/09/08