Medical science: Not all it’s cracked up to be

Me, I love science. New Scientist does too. So it makes us feel weird to point out that a lot of the time, it just doesn’t work:

many recent reports have raised the alarm that a shocking amount of the published literature in fields ranging from cancer biology to psychology is not reproducible.

Pharmaceuticals company Bayer, for example, recently revealed that it fails to replicate about two-thirds of published studies identifying possible drug targets (Nature Reviews Drug Discovery, vol 10, p 712).

Bayer’s rival Amgen reported an even higher rate of failure – over the past decade its oncology and haematology researchers could not replicate 47 of 53 highly promising results they examined (Nature, vol 483, p 531). Because drug companies scour the scientific literature for promising leads, this is a good way to estimate how much biomedical research cannot be replicated. The answer: the majority.

The author of that article, by the way, is trying to do something about the problem:

Here’s how it works. Scientists submit studies to us that they would like to see replicated. Our independent scientific advisory board – all members of which are leaders in their fields as well as advocates on the reproducibility problem – selects studies for replication. Service providers are then selected at random to conduct the experiments, and the results are returned to the original investigators, who can then publish them in a special issue of the open-access journal PLoS ONE. We will issue a “certificate of reproducibility” for studies that are successfully replicated.

In our pilot phase, we expect to attempt to replicate 40 to 50 studies. We also plan to publish an analysis of the overall success of what is essentially an experiment in reproducibility.

Still, a little weird, eh?