Donna Laframboise asks, "How many scientific papers just aren’t true? Enough that basing government policy on ‘peer-reviewed studies’ isn’t all it’s cracked up to be."
We’re continually assured that government policies are grounded in evidence, whether it’s an anti-bullying programme in Finland, an alcohol awareness initiative in Texas or climate change responses around the globe. Science itself, we’re told, is guiding our footsteps.
There’s just one problem: science is in deep trouble. Last year, Richard Horton, editor of the Lancet, referred to fears that ‘much of the scientific literature, perhaps half, may simply be untrue’ and that ‘science has taken a turn toward darkness.’
It’s a worrying thought. Government policies can’t be considered evidence-based if the evidence on which they depend hasn’t been independently verified, yet the vast majority of academic research is never put to this test. Instead, something called peer review takes place. When a research paper is submitted, journals invite a couple of people to evaluate it. Known as referees, these individuals recommend that the paper be published, modified, or rejected.
If it’s true that one gets what one pays for, let me point out that referees typically work for no payment. They lack both the time and the resources to perform anything other than a cursory overview. Nothing like an audit occurs. No one examines the raw data for accuracy or the computer code for errors. Peer review doesn’t guarantee that proper statistical analyses were employed, or that lab equipment was used properly. The peer review process itself is full of serious flaws, yet is treated as if it’s the handmaiden of objective truth.
And it shows. Referees at the most prestigious of journals have given the green light to research that was later found to be wholly fraudulent. Conversely, they’ve scoffed at work that went on to win Nobel prizes. Richard Smith, a former editor of the British Medical Journal, describes peer review as a roulette wheel, a lottery and a black box. He points out that an extensive body of research finds scant evidence that this vetting process accomplishes much at all. On the other hand, a mountain of scholarship has identified profound deficiencies.
. . .
Politicians and journalists have long found it convenient to regard peer-reviewed research as de facto sound science. Saying ‘Look at the studies!’ is a convenient way of avoiding argument ... We’ve long been assured that reports produced by the UN’s Intergovernmental Panel on Climate Change (IPCC) are authoritative because they rely entirely on peer-reviewed scientific literature. A 2010 InterAcademy Council investigation found this claim to be false, but that’s another story. Even if all IPCC source material did meet this threshold, the fact that one academic journal — and there are 25,000 of them — conducted an unspecified and unregulated peer review ritual is no warranty that a paper isn’t total nonsense.
If half of scientific literature ‘may simply be untrue’, then might it be that some of the climate research cited by the IPCC is also untrue? Even raising this question is often seen as being anti-scientific. But science is never settled. The history of scientific progress is the history of one set of assumptions being disproven, and another taking its place.
There's more at the link. Ms. Laframboise's full report may be read here (the link is to an Adobe Acrobat document in .PDF format).
This is precisely why I profoundly distrust any politician who tries to tell us that 'the science is settled'. All too often, it's far from settled. It may even be actively and deliberately fraudulent, producing results tailor-made to satisfy the objectives of those who've funded the research. Too many 'researchers' begin with a goal in mind, their conclusions already identified, and then seek evidence that will substantiate what they want to prove. Anything to the contrary is ignored or discarded, or flagrantly manipulated to achieve the desired result (as in this example, to cite just one - there are many more).
That's not research at all. It's pseudo-scientific sleight of hand. It's a shell game.