A Non-scientist’s Guide to Checking the Science Behind a Claim.
 |
Let’s check the science: How can I check the science myself? |
I began the “Let’s check the science” series on this website to encourage non-scientist visitors to do their own checking. The idea was that if I supplied a few examples, they would see how it can be done and try it themselves. I now realize that was probably wishful thinking (well, we all do it, don’t we?), so I decided a fully-fledged guide was in order. It wasn’t an easy task, and I’m expecting to revise it, probably many times. This is version 1.4. Feedback would be welcome. You can access a printable pdf version here.
| This guide assumes you are happy to accept science as the best way of discovering the truth about the natural world. If that’s not the case for you, why not have a look at Trusting the science first? |
Be prepared. |
|
| Know something about the scientific process. | Browse the Hallmarks of science at Science Or Not? |
| Know something about the tactics used to mislead. | Browse the Science red flags at Science Or Not? |
| Look at some examples of the checking process. | Browse Let’s check the science at Science Or Not? |
The promoters’ arguments – science or red flags? |
|
| Locate sources of the claim. | Claims you’ve heard about or seen on TV aren’t very helpful. You need a permanent version to check the details. Find printed sources or use a search engine to find web pages, documents, podcasts or videos. |
| Disregard any headlines. | Headlines are often misleading. Ignore them. |
| Clarify the claim. | Identify exactly what is being claimed. Summarise the main points. |
| Note the language. | What’s the tone of the language? Emotive? Confrontational? Adversarial? Provocative? Or calm, rational, logical, technical? |
| Look for Science red flags. | Identify any logical fallacies and cognitive biases in the information. |
Being skeptical. |
|
| Evaluate the source of information. | The most trustworthy information comes from educational institutions, government organisations, scientific bodies and journals and globally-respected sources such as some encyclopaedias. Wikipedia is not bad. Be sceptical of all other sources. |
| Evaluate the type of publication. | Peer reviewed papers or systematic reviews are most reliable. Case studies and conference papers less so. Press releases and media reports can have lots of spin – marketing material even more so. Be particularly wary of material that is anti-science (usually identified by conspiracy theories directed at the scientific establishment). |
| Evaluate the expertise of the author. | If the article is anonymous, disregard it.Does the author have expertise in the area? You can Google the author’s name to find out how respected they are in the field. Disregard anyone who is giving an opinion outside their area of expertise.Are contact details given? You may be able to contact the author to confirm expertise. |
| Think about vested interests and biases. | Anyone who publishes information has a motive – but is that bias distorting the facts? Could the author or organisation behind the article have a conflict of interest? Is there an attempt to sell something? Is there an obvious ideological bias? |
| Weigh up the expertise, red flags and any hallmarks. | Considering all the above, decide how trustworthy the information is. Make a list of all the details of the claim that need checking on. |
The scientific evidence. |
|
| Do a web search. | Use a search engine with keywords particular to the claim. |
| Select trustworthy sources. | Look for web pages you can trust in the search results. URLs that contain .edu, .gov or .org are good indicators. Wikipedia is a reasonable starting point and sometimes sites like Snopes.com. But remember that these are secondary sources. Preferably use them only to find links to primary sources. A list of journals that should not be trusted can be found at Scholarly Open Access. |
| Locate primary sources. | Look in your search results or secondary sources for links to original scientific papers. Google Scholar can help here, and Unsourced is useful for press articles in the UK. Try to find papers from reliable journals (see the Wikipedia List of scientific journals). Don’t expect to find a study that specifically answers your question. |
| Look for systematic reviews. | Systematic reviews are valuable because they survey the literature and examine all papers on a particular topic. Good systematic reviews will have a ‘plain language’ summary. You can access reviews on health at sites such as The Cochrane Library, The Joanna Briggs Institute, Systematic Reviews, and The Campbell Collaboration. |
| What to do about paywalls. | When papers are behind paywalls, usually only the Abstract can be read for free. It’s dangerous to rely on abstracts because they often contain some spin. If you can’t access the full paper via membership of an educational institution or library, Google the title and authors. You may be lucky enough to find a free version somewhere. The Open Access Button could be helpful too. |
| Read as much of each paper as you can. | Definitely read the Abstract and Conclusion; preferably the Discussion as well. You should at least scan the Methods and Results to make sure empirical evidence actually exists, but they are often very technical. |
| Look for retractions and rebuttals. | If a paper seems doubtful, check at Retraction Watch to see whether it has been retracted. If you install the rbutr extension on your web browser, you may be able to use it to find rebuttals of the paper. |
| Rely on reliable secondary sources only when unavoidable. | Use these if there are no primary sources, or the primary sources are too technical for you to follow. Make sure these sources are reliable (as explained above) |
| Ask scientists. | There are websites where you can put questions directly to scientists. Use a search engine to find them, but make sure they are trustworthy. |
DIY evidence. |
|
| Carry out your own test. | This is only rarely possible, and because it won’t be a proper controlled experiment, regard it only as a preliminary investigation. |
Conclusion. |
|
| Do the studies show a mechanism or only correlation? | Epidemiological studies (RCTs) usually show only correlation, which does not prove cause and effect. More basic research is usually needed to identify a cause or a mechanism. |
| Evaluate all the evidence. | Ask yourself these questions: Does the evidence seem to be credible? Does it all fit together? What does this mean for the original claim? |
| Accept or reject. | Do you accept the claim or not? Or is it perhaps partly true? Remember to guard against wishful thinking and other red flags. |
| How conditional is your decision? | How confident are you of your conclusion? What evidence would you need to change your mind? Don’t fall for single-study syndrome. |
Update 2014/02/16: In this workshop from TAM 2012, Steven Novella, David Gorski, Harriet Hall, and Rachael Dunlop tell you how to find reliable health information online and skeptically evaluate the information you find.
Other useful guides
A Rough Guide to Spotting Bad Science from Compound Interest.
How can I find accurate science information on the internet? from Science Questions with Surprising Answers.
Evaluating Web Pages: Techniques to Apply & Questions to Ask from UC Berkeley.
Evaluating Web Content from State University of New York
Evaluation Clues for Articles Found on the Web or in Library Databases from Colorado State University.
How to Evaluate Journal Articles from Colorado State University.
How to Evaluate a Scientific Research Article from Optometry Today.
Critical Appraisal of Scientific Articles from Deutsches Ärzteblatt International.
Evaluating Treatment Claims: A Primer by Harriet Hall at Science-Based Medicine.
How to evaluate the quality of scientific research by Skeptical Raptor.
8 ways to be a more savvy science reader by Susannah Locke.
How to Conduct Scientific Research On the Internet (Without Getting Duped) by Alan Henry
10 steps for evaluating scientific papers by Fallacy Man at The Logic of Science
How to read and understand a scientific paper: a guide for non-scientists by Jennifer Raff at the LSE Impact Blog.
|
This is one of ScienceOrNot’s Let’s check the science series. |
Science or not? 
One of the points is “Ask Scienctists: There are websites where you can put questions directly to scientists. Use a search engine to find them, but make sure they are trustworthy.” Are you able to list some trustworthy, good websites?
Some suggestions, Tanya. I’ve never used any of these websites, so I can’t vouch for them, but they are from reasonably reputable sources:
ABC Science – Ask an expert
National Earth Science Teachers Association – Windows to the Universe
Union of Concerned Scientists
Howard Hughes Medical Institute
Argonne National Laboratory – Newton
UCSB Science Line
Cornell Center for Materials Research
You’d need to decide which site is most appropriate for each question you want to ask. Some of them are limited to students, or their parents or teachers.
Thanks Graham. I had no idea such sites existed.
You’re arrogant. You have become paralyzed by science. Bruce Lee could think better than you can.
Science is the new religion.
Ah, the good old ad hominem: “scientists are arrogant”. Stay tuned for a future red flag on this ploy. (Update 2012/02/06: The post on this red flag can now be found here).
Can we look forward to some evidence to back up your accusations, Sven?
YOU are using this word INCORRECTLY. Google “Plover ad hominem.” ALL I did to you was an insult- that is it.
Scientists ARE arroganrt. My evidence is that human knowledge is extremely lmited. Humans MINDS are extremely limited. Science is a HUMAN endeavor and extremely lmited. It all hinges on the ability of our MINDS.Science is no better than humans’ abaility to understand the world.
Everything could be wrong, but not necessarily.
‘Scientists ARE arroganrt’ – my arrogance meter just exploded, I’d better get back to my lab and fix it.
Philosophy trumps science in matters science cannot address such as God and the meaning of our existence. Science does not even have the ABILITY to address the most important questions in life.
Sven, you haven’t the slightest idea of what evidence is. In the article above, I went to considerable length to show why science is the best method we have of finding out about the real world. In my book, that’s not arrogance. You, on the other hand, expect people to accept your empty edicts simply because you assert them. That’s arrogance.
As I’ve written in response to another of your comments, I think you’ll find that science is not interested in addressing the question of gods.
Reblogged this on Utopia – you are standing in it!.
On evaluating expertise. You fail to consider that an intelligent and interested person might have great expertise in some area, even if it is not their profession and they do not have “respect in the field”. It is most unfair to automatically dismiss such a person. They should be judged on the quality of their arguments and the evidence that supports them (or, to be properly Scientific a la Popper, on the failure of strenuous attempts to disprove them)
The biggest mistake that fake scientists make is pretending that Science demonstrates what is true. It does not. Science treats all truths as partial or conditional, and open to disproof by either the failure of one or more of its predictions, or its inability to account for known phenomena. And that is true despite the tendency of some scientists to hang on to some theories long after those theories have been discredited. What is more, older theories can still be useful within their limitations. That is a good job, because the maths of Newtonian Mechanics is relatively simple, whereas Relativity and Quantum Mechanics are only properly accessible to superior mathematicians.
Tom, I think you’ve overlooked the fact that this is a guide for non-scientists, that is, people who probably don’t have the background to judge the quality of the arguments and the supporting evidence. In that case they need to look for more indirect indications that the word of the “expert” can be taken as reliable.
No, I have not overlooked that at all. The main points I am making is that it excludes good and correct information from people that are not specialists in the field
Furthermore “respect in the field” is a poor criterion for deciding who to trust, and makes many charlatans and deluded types seem trustworthy
The other criteria that you suggest are more objective and would not exclude contributions from an intelligent, logical, honest and well-informed person that has not made a career of the subject being debated, yet is more trustworthy than many professionals in the field.