Replication Problems

We tried to publish a replication in Science.

We believe that it is bad policy for journals like Science to publish big, bold ideas and then leave it to subfield journals to publish replications showing that those ideas aren’t so accurate after all. Subfield journals are less visible, meaning the message often fails to reach the broader public. They are also less authoritative, meaning the failed replication will have less of an impact on the field if it is not published by Science.

Open and transparent science can only happen when journals are willing to publish results that contradict previous findings. We must resist the human tendency to see a failed replication as an indication that the original research team did something wrong or bad. We should continue to have frank discussions about what we’ve learned over the course of the replication crisis and what we could be doing about it (a conversation that is currently happening on Twitter).

It is quite amazing that - if the account by those authors is accurate - a scientific journal should agree that the conclusions of a paper it published had been shown to be wrong, but refuse to do anything about it. I'd have thought at the bare minimum it should have offered to publish a note acknowledging the failed replication.

It is quite amazing that - if the account by those authors is accurate - a scientific journal should agree that the conclusions of a paper it published had been shown to be wrong, but refuse to do anything about it. I'd have thought at the bare minimum it should have offered to publish a note acknowledging the failed replication.

But presumably it couldn't even publish an acknowledgement until after the peer review process - something which was also refused.

But presumably it couldn't even publish an acknowledgement until after the peer review process - something which was also refused.

Yes. Though another odd feature of the response is that - according to the authors - the journal "concluded that we had offered a conclusive replication of the original study," even without sending the paper for review. It seems odd all round.

It is quite amazing that - if the account by those authors is accurate - a scientific journal should agree that the conclusions of a paper it published had been shown to be wrong, but refuse to do anything about it. I'd have thought at the bare minimum it should have offered to publish a note acknowledging the failed replication.

A failed replication doesn't necessarily mean anything beyond that the second team failed to replicate the experiment. It doesn't disprove the initial study's positive results, however it may hint at a study having validation issues.

A failed replication doesn't necessarily mean anything beyond that the second team failed to replicate the experiment. It doesn't disprove the initial study's positive results, however it may hint at a study having validation issues.

I'm no scientist but it was my understanding that if something is failed to be reproduced in an experiment, then the first study is put into doubt, no?

A failed replication doesn't necessarily mean anything beyond that the second team failed to replicate the experiment. It doesn't disprove the initial study's positive results, however it may hint at a study having validation issues.

 
In this case they reported three replications, once of which was a direct replication whose statistical power - the probability of finding an effect of the size previously reported, if it's there - was 80%. So they'd have had to be quite unlucky to miss it. The draft paper can be found here:
https://osf.io/86xhp/

I'm no scientist but it was my understanding that if something is failed to be reproduced in an experiment, then the first study is put into doubt, no?

The first experiment should never be taken at face value as something consistent. An interesting find without replication is just that. We used to know this, but science journalism -especially in mainstream publications- has made the headline its real star... And there is nothing glamorous about retraction.

I'm no scientist but it was my understanding that if something is failed to be reproduced in an experiment, then the first study is put into doubt, no?

I suppose I could have been more clear. Yes it produces doubt, but no it is not a disproof. To disprove it they would need to find the methodological flaws within the study that would have lead to a positive result where replications failed to find that same positive result.

 
In this case they reported three replications, once of which was a direct replication whose statistical power - the probability of finding an effect of the size previously reported, if it's there - was 80%. So they'd have had to be quite unlucky to miss it. The draft paper can be found here:
https://osf.io/86xhp/

I would certainly say three failed replications casts three times as much doubt upon the original study. The experiment itself is a simple test as well, but I wonder how the original experiment went wrong.

The whole replication crisis certainly gives me misgivings about the field of science as a whole, that major journals rarely have the ethical fortitude to publish negative results let alone fund replications is quite disturbing.