To paraphrase George Orwell “Some evidence is more equal than others”. Where evidence is concerned there is a distinct heirarchy with weak forms such as case reports and anecdotes (stories) at the bottom and randomised double blind placebo controlled trials (RDBPCTs) and metanalyses at the top, regarded as the most robust.
There is also the matter of where the evidence is published. A survey about alleged vaccine reactions published in “Fave Celebs” magazine is far less trustworthy than a paper on the same subject published in the British Medical Journal. The best journals operate a process of “Peer Review” where any paper which is submitted for publication is looked at by specialists in the field under study and also by statisticians to make sure the maths has been done properly and the conclusions are justified.
“That's great”, you may be thinking, “very simple, now all I need to do is look at peer reviewed DBPCTs and ignore everything else”. Well, unfortunately not, like so much else in life, it’s not that simple. In fact the act of looking at evidence has become a branch of science in its own right these days (see references below) and there are distinct dos and don'ts when it comes to reading and interpreting papers correctly.
For instance, a trial can claim to be blinded but if the authors don't explain the methods used to achieve this you would be right to be sceptical, and comparing two treatments is going to give less reliable results if the patients being studied know whether they're getting the real treatment or not. The same for randomization -
Peer review also isn't always everything it might be. Take homeopathic journals for instance. Many will claim to be peer reviewed but on closer inspection it turns out that it is other homeopaths who are doing the reviewing, not necessarily experts in the condition being studied or statisticians or people with an expertise in trial design. There would have to be a strong suspicion of publication bias if the people doing the reviewing are already convinced that homeopathy is a valid method of treatment and in fact this is supported by the evidence when such journals are analysed.
Even once a paper is published there are ways of presenting it which can make it look as if something has been shown when it hasn't. Homeopaths and proponents of CAM and other pseudoscience such as the raw feeding of pets are particularly adept at this. They will selectively quote from parts of a paper while ignoring other, more significant portions thus “spinning” the actual conclusions. Trials which conclude that homeopathy is ineffective will be either ignored or attacked on bogus grounds such as being unhomeopathic while at the same time papers which appear to the outsider as equally 'unhomeopathic', breaking multiple of the so called 'laws of homeopathy' are accepted, so long as the findings also support the idea that homeopathy works.
This is known as 'cherry picking' -
So, here are a few general points to bear in mind when considering evidence about CAVM, which homeopaths and others with similar, single agenda outlooks usually choose to forget:
1/ The purpose of a trial is to test a theory, not to prove it. The scientific process at its best is a way of impartially testing theories while correcting for the problems of subjective assessment and cognitive bias that we are all prone to, particularly if we feel strongly about the outcome. Researchers such as homeopaths who know what outcome they want before they start and are already convinced that homeopathy is effective do not make the most impartial investigators of homeopathy.
2/ The above notwithstanding, the RDBPCT is not always the be all and end all when it comes to testing drugs. Despite what is often claimed by detractors of real medicine some drugs have an effect that is so obvious that there is no need for such a trial. To quote veterinary surgeon Dr Morag Kerr 'the RDBPCT isn't by any means the "gold standard" test to be applied to every therapeutic intervention. It's quite pointless and indeed highly unethical to go through such an exercise for a self-
CAM proponents are very quick to point out that some drugs and other treatments have never had a RDBPCT performed on them, but without mentioning the reason why -
3/ Homeopathy most certainly does NOT use "small doses of various substances" (as claimed here for example -
All this of course is like water off the proverbial duck's back to CAM believers. They're not interest in robust methodologies and protocols or detailed analysis of statistics, they are intersted purely and simply in results. It doesn’t matter how dreadful a trial is or even if it’s just an open questionnaire, by their definition a good trial is simply one which gives results they want. while anything which contradicts their world view is to be roundly condemned.
Now, however you know enough to disabuse them of this misconception, and when told there are hundreds of trials which “prove” that homeopathy works you can nod politely in agreement before pointing out that, while strictly correct, this statement is somewhat marred by the fact that virtually none of them are worth the paper they're printed on!
Click on the links below if you’d like to read more on the subject of how to interpret a scientific study and remember -
Evidence based medicine websites:
Novella (2007) How Much Modern Medicine is Evidence-
How to interpret papers:
Shady research is rampant: