Science uses peer review

There’s no peak scientific body that judges which models are right and which are wrong. It all depends on peer review and reasoned consensus.

This is one of ScienceOrNot’s Hallmarks of science. See them all here.

In short…

Scientists present new and important findings to the scientific community, where other scientists working in the field scrutinize them.

Peer review keeps the official scientific literature reasonably honest and factually reliable. … But it does not pretend to eliminate error, nor does it guarantee certainty or truth.

John Ziman, British physicist and philosopher, 2000

What is peer review?

Peer review is science’s method of quality control. New scientific work is exposed to the scrutiny of other scientists, who look for inconsistencies and errors. The most important part of peer review comes before a scientific paper is published.

How peer review happens.

Peer review often starts at an early stage, when scientists speak about their work at conferences. Their peers in the audience may have criticisms and suggestions at this stage. This feedback helps when the scientists eventually write a formal report of their work for publication.

Before a scientific report is published by a reputable journal, the editor sends it to a selection of acknowledged experts in the field – peers of the report authors. They check to make sure the work is original, significant and valid. These experts may recommend changes to the report, or even rejection. The authors may then incorporate their suggestions into the report before it is published. This process is called pre-publication peer review (and is generally what scientists mean when they talk of ‘peer review’).

After a paper is published, it may be read by a wide section of the scientific community. There may be more comments and feedback, some of which is published in later issues of the journal, or other journals. This is post-publication peer review. Increasingly, this type of review is taking place in online forums, which allows the value of reports to be indicated very quickly.

Update 2013/11/09: Mark Lorch explains just how important post-publication peer review is becoming here.

A peer review process is also used in deciding research grants. Proposals for research programs from scientists are examined by a panel of their peers. The panel makes decisions about which research is most deserving of the funds available.

Why peer review is needed.

Pre-publication peer review ensures that published reports are reliable, reasonably accurate and that the work has been carried out according to proper scientific practice. It does not guarantee that the conclusions are true. Continued review, as scientists replicate the work, check details and carry out further investigations, may support the conclusions, or it may show that they are wrong. (see All scientific models are tentative)  A study that has withstood extensive post-publication review can be regarded as very reliable.

Because peer review means that the work of researchers will be scrutinised by respected scientists in their field, the process encourages a sense of scepticism and caution in the researchers.

Pre-publication peer review cannot eliminate scientific fraud. Reviewers must assume results as described in the paper are honest – they do not have the time to check these. But fraud will sooner or later be detected by post-publication review, as scientists find they cannot replicate the findings of the original study.

Bogus science tends to avoid peer review.

Examples

• Examples of scientific journals covering all scientific disciplines are Nature, Science and PLoS ONE. There are thousands of journals covering more specific fields, for example The Astrophysical Journal, Journal of the American Chemical Society, Geophysical Research Letters, Advances in Physics, Journal of Molecular Biology, Ecology, Theoretical and Applied Genetics, The Journal of Neuroscience, and The Lancet.
• The Australian Research Council publishes a list of scholarly, peer reviewed journals that publish original research.
• In 1913, Philosophical Magazine published a paper called ‘On the Constitution of Atoms and Molecules’  by Niels Bohr. It described a new model for atoms which incorporated both earlier models and the emerging ideas of quantum theory. Although the Bohr model has been refined by later work, it is still the foundation of modern atomic models.
• In 1998, The Lancet published a paper by Dr. Andrew Wakefield and others. The paper reported on intestinal disorders found in autistic children and strongly suggested that these disorders had resulted from the measles-mumps-rubella (MMR) vaccine. Despite the lack of evidence from other studies to support this link, Wakefield’s claims were widely reported in the media, resulting in a global movement away from vaccination which had serious health implications. After an investigation by journalist Brian Deer, published by the British Medical Journal, and an inquiry conducted by the UK General Medical Council,  The Lancet  retracted the paper in 2010.
• In 2002, researchers from Stockholm University and the Swedish National Food administration announced that they had discovered high levels of acrylamide, a chemical thought to cause cancer, in many common foods. They went public with their findings before  publishing them in a scientific journal because they believed their was an immediate threat to public safety. Previous and subsequent peer-reviewed studies have mostly failed to confirm such a link (see here, here, here, here, here, here, here and here).
• In late 2011, the editor of the journal Remote Sensing resigned. The journal had published a paper which cast doubt on the model of human-induced climate change. The editor explained that post-publication review had shown that there were errors of methodology and false claims in the paper, that the pre-publication review had been faulty and that the paper should not have been published.

Further reading

Peer-reviewed literature: What does it take to publish a scientific paper? at The Logic of Science.

John Ziman’s quote is from Real Science, p43
The Peer review image is from the University of Delaware.
The Peer review diagram is based on a diagram in ‘Peer Review and the Acceptance of New Scientific Ideas’ from Sense About Science (p34).

This is one of ScienceOrNot’s Hallmarks of science. See them all here.

This page reviewed and updated: 2013/09/19