Jullia Belluz, common-sense and evidence-oriented journalist (known to me from her great "Science-ish" Maclean's column) and Trudeau Scholar and Assistant Professor of Law at the University of Ottawa, Steven Hoffman, team up in their Burden of Proof column for Vox.
This week, in "The one chart you need to understand any health study" they help readers with a simple approach to understanding how to evaluate levels of evidence. Not all research is created equally:
I think the chart it is a good start, and I wish it were as simple as this. Some sneaky (or inept) researchers are good at making trials look randomized, blinded, and so on but the controls, conflicts of interest, low study numbers, etc. mean that the data they gather is not very useful at all. Sometimes, the way the papers are written, it's easy to think of the conclusion as groundbreaking and accurate, but digging deeper into the methods it becomes clear that the authors did a little.... 'creative interpretation'.
Even the highest form of evidence comes in different flavours:
Not all systematic reviews are created equally, either.
And while some evidence is stronger than other evidence, it doesn't necessarily mean anything when it comes to applying it to you, the individual. Fortunately, Ms Belluz and Mr Hoffman get it.
Even with the best available evidence from around the world at our disposal, we have to analyze it and apply it to our particular circumstances. A personal experience with the success or failure of a drug, like an allergic reaction, is more informative for you than the most rigorous study on the drug ever could be.
It can be challenging to spot issues with quality amongst the jargon and statistics. It is so refreshing to see journalists like Julia Belluz who get this and who are raising the bar for colleagues to be responsible with their science reporting.