The danger of anecdotal evidence
Anecdotal evidence is one of the most common forms of evidence. We encounter it frequently in life, from those stories about the relative who tried some new cure to reports of children developing autism after receiving vaccines to stories on Fox News about how some snowstorm somehow disproves global warming. However common, anecdotal evidence is the worst and most misleading form of evidence.
First, anecdotes by their very nature tend to have a sample size of one. You cannot tell much from a single data point. You cannot tell if the results were due to chance or determine the side effects of a treatment. You have no idea how the placebo effect impacted the results. In controlled medical studies, 50% or more of a control group receiving a sugar pill will show improvement along the lines of that expected for the drug that is being tested along with side effects (i.e. hair loss if testing a new chemotherapy drug). Without more data, there's no way to tell if any improvement is actually due to a drug or treatment or happened due to sheer coincidence.
Second, humans have a bias toward reporting positive results. What that means in anecdotal evidence is you're far more likely to hear about someone improving than you are about someone who either remained the same or got worse. As a result, a treatment or technique will sound far more effective than it actually is.
Third, confirmation bias is a major problem with anecdotes. Humans are far more likely to report—and remember—results that confirm personal ideas. If you're a science denier, weather stations that show a cooling trend will appear to be far more numerous than they really are.
Finally, cherry-picking the data is rampant with anecdotes. In many cases with science deniers, they'll deliberately manipulate the data to give a false impression (i.e. most of the graphics you find on science denying sites like C3 Headlines).
So there you have it. Anecdotal evidence alone is at best misleading and usually bogus. Without support from hard data and experiments, it's really a form of evidence that is most suitable for con artists and charlatans. It's suitable mostly to illustrate findings from more rigorous studies and little else. The problems occur when all someone has is anecdotal evidence.
What triggered this rant? Easy. I've spent quite a bit of time over the past few months pouring over various alternate medicine books that family members had read and were quite convinced by. Those books were rife with anecdotal evidence (i.e. Treated patient X for condition Y with ______ and _______ and their condition improved) and nonexistent on evidence from controlled randomized studies. And the authors of the books just happen to sell all the various dietary supplements you need to start their program. What amazed me is that these particular family members have undergraduate degrees in science—and still completely missed what to me were obvious red flags: No clinical studies using randomized control groups, the financial incentive of the authors to push their various programs, reliance entirely on anecdotes, advocating extreme diets while replacing the lost nutrients with expensive supplements, etc.
What can be done about anecdotal evidence? Not much, really. I've explained what anecdotal evidence is, why it's the worst form of evidence, and point the way toward studies that were done with much more data and better design. But the old adage applies: You can lead a horse to water but you can't make the horse drink.
First, anecdotes by their very nature tend to have a sample size of one. You cannot tell much from a single data point. You cannot tell if the results were due to chance or determine the side effects of a treatment. You have no idea how the placebo effect impacted the results. In controlled medical studies, 50% or more of a control group receiving a sugar pill will show improvement along the lines of that expected for the drug that is being tested along with side effects (i.e. hair loss if testing a new chemotherapy drug). Without more data, there's no way to tell if any improvement is actually due to a drug or treatment or happened due to sheer coincidence.
Second, humans have a bias toward reporting positive results. What that means in anecdotal evidence is you're far more likely to hear about someone improving than you are about someone who either remained the same or got worse. As a result, a treatment or technique will sound far more effective than it actually is.
Third, confirmation bias is a major problem with anecdotes. Humans are far more likely to report—and remember—results that confirm personal ideas. If you're a science denier, weather stations that show a cooling trend will appear to be far more numerous than they really are.
Finally, cherry-picking the data is rampant with anecdotes. In many cases with science deniers, they'll deliberately manipulate the data to give a false impression (i.e. most of the graphics you find on science denying sites like C3 Headlines).
So there you have it. Anecdotal evidence alone is at best misleading and usually bogus. Without support from hard data and experiments, it's really a form of evidence that is most suitable for con artists and charlatans. It's suitable mostly to illustrate findings from more rigorous studies and little else. The problems occur when all someone has is anecdotal evidence.
What triggered this rant? Easy. I've spent quite a bit of time over the past few months pouring over various alternate medicine books that family members had read and were quite convinced by. Those books were rife with anecdotal evidence (i.e. Treated patient X for condition Y with ______ and _______ and their condition improved) and nonexistent on evidence from controlled randomized studies. And the authors of the books just happen to sell all the various dietary supplements you need to start their program. What amazed me is that these particular family members have undergraduate degrees in science—and still completely missed what to me were obvious red flags: No clinical studies using randomized control groups, the financial incentive of the authors to push their various programs, reliance entirely on anecdotes, advocating extreme diets while replacing the lost nutrients with expensive supplements, etc.
What can be done about anecdotal evidence? Not much, really. I've explained what anecdotal evidence is, why it's the worst form of evidence, and point the way toward studies that were done with much more data and better design. But the old adage applies: You can lead a horse to water but you can't make the horse drink.
Comments
Post a Comment