Science is mired in a “replication” crisis. Fixing it will not be easy.

Science is mired in a “replication” crisis. Fixing it will not be easy. 

by Andrew Gelman

If something at random does turn up and achieve statistical significance, it is likely to be a massive overestimate of any true effect. In an attempt at replication, we’re likely to see something much closer to zero.

The failed replications have been no surprise to many scientists, including myself, who have lots of experience of false starts and blind alleys in our own research.

The big problem in science is not cheaters or opportunists, but sincere researchers who have unfortunately been trained to think that every statistically “significant” result is notable.

When you read about research in the news media (and, as a taxpayer, you are indirectly a funder of research, too), you should ask what exactly is being measured, and why.

If scientists want to regain public trust I'd suggest a quality review of the varied fields including parapsychology.

Two thoughts. First is the idea that initial conditions for a study may be the problem and not the protocol. For instance, cold fusion worked, but then it didn't. I spoke with one of the later researchers who indicated that the problem is in the palladium alloy. The wrong mix of impurities stops the reaction. Buying palladium for replication studies does not assure replication.

We see this Instrumental TransCommunicaiton (ITC). When people replicate an old experiment, they typically do so with more contemporary equipment which is functionally the same but materially different. They seldom successfully replicate the study. Old equipment used the now hard to find germanium transistors.

My second thought is that people used for replication are the problem. Many academics attempt to replicate experiments using college students as practitioners. In ITC, that is like proving that Mickael Jorden is a fraud by using average people to replicate his scoring.

There is quite a lot of study about the experimenter effect. Here, I am not talking about honest versus dishonest scientists. I am talking about the influence of the scientist's expectations on the outcome. A doubting researcher is more apt to have a "failure to replicate" outcome.