A Non-scientist’s Guide to Checking the Science Behind a Claim.
|Let’s check the science: How can I check the science myself?|
I began the “Let’s check the science” series on this website to encourage non-scientist visitors to do their own checking. The idea was that if I supplied a few examples, they would see how it can be done and try it themselves. I now realize that was probably wishful thinking (well, we all do it, don’t we?), so I decided a fully-fledged guide was in order. It wasn’t an easy task, and I’m expecting to revise it, probably many times. This is version 1.4. Feedback would be welcome. You can access a printable pdf version here.
|This guide assumes you are happy to accept science as the best way of discovering the truth about the natural world. If that’s not the case for you, why not have a look at Trusting the science first?|
|Know something about the scientific process.||Browse the Hallmarks of science at Science Or Not?|
|Know something about the tactics used to mislead.||Browse the Science red flags at Science Or Not?|
|Look at some examples of the checking process.||Browse Let’s check the science at Science Or Not?|
The promoters’ arguments – science or red flags?
|Locate sources of the claim.||Claims you’ve heard about or seen on TV aren’t very helpful. You need a permanent version to check the details. Find printed sources or use a search engine to find web pages, documents, podcasts or videos.|
|Disregard any headlines.||Headlines are often misleading. Ignore them.|
|Clarify the claim.||Identify exactly what is being claimed. Summarise the main points.|
|Note the language.||What’s the tone of the language? Emotive? Confrontational? Adversarial? Provocative? Or calm, rational, logical, technical?|
|Look for Science red flags.||Identify any logical fallacies and cognitive biases in the information.|
|Evaluate the source of information.||The most trustworthy information comes from educational institutions, government organisations, scientific bodies and journals and globally-respected sources such as some encyclopaedias. Wikipedia is not bad. Be sceptical of all other sources.|
|Evaluate the type of publication.||Peer reviewed papers or systematic reviews are most reliable. Case studies and conference papers less so. Press releases and media reports can have lots of spin – marketing material even more so. Be particularly wary of material that is anti-science (usually identified by conspiracy theories directed at the scientific establishment).|
|Evaluate the expertise of the author.||If the article is anonymous, disregard it.Does the author have expertise in the area? You can Google the author’s name to find out how respected they are in the field. Disregard anyone who is giving an opinion outside their area of expertise.Are contact details given? You may be able to contact the author to confirm expertise.|
|Think about vested interests and biases.||Anyone who publishes information has a motive – but is that bias distorting the facts? Could the author or organisation behind the article have a conflict of interest? Is there an attempt to sell something? Is there an obvious ideological bias?|
|Weigh up the expertise, red flags and any hallmarks.||Considering all the above, decide how trustworthy the information is. Make a list of all the details of the claim that need checking on.|
The scientific evidence.
|Do a web search.||Use a search engine with keywords particular to the claim.|
|Select trustworthy sources.||Look for web pages you can trust in the search results. URLs that contain .edu, .gov or .org are good indicators. Wikipedia is a reasonable starting point and sometimes sites like Snopes.com. But remember that these are secondary sources. Preferably use them only to find links to primary sources. A list of journals that should not be trusted can be found at Scholarly Open Access.|
|Locate primary sources.||Look in your search results or secondary sources for links to original scientific papers. Google Scholar can help here, and Unsourced is useful for press articles in the UK. Try to find papers from reliable journals (see the Wikipedia List of scientific journals). Don’t expect to find a study that specifically answers your question.|
|Look for systematic reviews.||Systematic reviews are valuable because they survey the literature and examine all papers on a particular topic. Good systematic reviews will have a ‘plain language’ summary. You can access reviews on health at sites such as The Cochrane Library, The Joanna Briggs Institute, Systematic Reviews, and The Campbell Collaboration.|
|What to do about paywalls.||When papers are behind paywalls, usually only the Abstract can be read for free. It’s dangerous to rely on abstracts because they often contain some spin. If you can’t access the full paper via membership of an educational institution or library, Google the title and authors. You may be lucky enough to find a free version somewhere. The Open Access Button could be helpful too.|
|Read as much of each paper as you can.||Definitely read the Abstract and Conclusion; preferably the Discussion as well. You should at least scan the Methods and Results to make sure empirical evidence actually exists, but they are often very technical.|
|Look for retractions and rebuttals.||If a paper seems doubtful, check at Retraction Watch to see whether it has been retracted. If you install the rbutr extension on your web browser, you may be able to use it to find rebuttals of the paper.|
|Rely on reliable secondary sources only when unavoidable.||Use these if there are no primary sources, or the primary sources are too technical for you to follow. Make sure these sources are reliable (as explained above)|
|Ask scientists.||There are websites where you can put questions directly to scientists. Use a search engine to find them, but make sure they are trustworthy.|
|Carry out your own test.||This is only rarely possible, and because it won’t be a proper controlled experiment, regard it only as a preliminary investigation.|
|Do the studies show a mechanism or only correlation?||Epidemiological studies (RCTs) usually show only correlation, which does not prove cause and effect. More basic research is usually needed to identify a cause or a mechanism.|
|Evaluate all the evidence.||Ask yourself these questions: Does the evidence seem to be credible? Does it all fit together? What does this mean for the original claim?|
|Accept or reject.||Do you accept the claim or not? Or is it perhaps partly true? Remember to guard against wishful thinking and other red flags.|
|How conditional is your decision?||How confident are you of your conclusion? What evidence would you need to change your mind? Don’t fall for single-study syndrome.|
Update 2014/02/16: In this workshop from TAM 2012, Steven Novella, David Gorski, Harriet Hall, and Rachael Dunlop tell you how to find reliable health information online and skeptically evaluate the information you find.
Other useful guides
A Rough Guide to Spotting Bad Science from Compound Interest.
How can I find accurate science information on the internet? from Science Questions with Surprising Answers.
Evaluating Web Pages: Techniques to Apply & Questions to Ask from UC Berkeley.
Evaluating Web Content from State University of New York
Evaluation Clues for Articles Found on the Web or in Library Databases from Colorado State University.
How to Evaluate Journal Articles from Colorado State University.
How to Evaluate a Scientific Research Article from Optometry Today.
Critical Appraisal of Scientific Articles from Deutsches Ärzteblatt International.
Evaluating Treatment Claims: A Primer by Harriet Hall at Science-Based Medicine.
How to evaluate the quality of scientific research by Skeptical Raptor.
8 ways to be a more savvy science reader by Susannah Locke.
10 steps for evaluating scientific papers by Fallacy Man at The Logic of Science
|This is one of ScienceOrNot’s Let’s check the science series.|