It is quite amazing that - if the account by those authors is accurate - a scientific journal should agree that the conclusions of a paper it published had been shown to be wrong, but refuse to do anything about it. I'd have thought at the bare minimum it should have offered to publish a note acknowledging the failed replication.
(2019-07-02, 06:56 AM)Chris Wrote: [ -> ]It is quite amazing that - if the account by those authors is accurate - a scientific journal should agree that the conclusions of a paper it published had been shown to be wrong, but refuse to do anything about it. I'd have thought at the bare minimum it should have offered to publish a note acknowledging the failed replication.
But presumably it couldn't even publish an acknowledgement until after the peer review process - something which was also refused.
(2019-07-02, 07:13 AM)Typoz Wrote: [ -> ]But presumably it couldn't even publish an acknowledgement until after the peer review process - something which was also refused.
Yes. Though another odd feature of the response is that - according to the authors - the journal "concluded that we had offered a conclusive replication of the original study," even without sending the paper for review. It seems odd all round.
(2019-07-02, 06:56 AM)Chris Wrote: [ -> ]It is quite amazing that - if the account by those authors is accurate - a scientific journal should agree that the conclusions of a paper it published had been shown to be wrong, but refuse to do anything about it. I'd have thought at the bare minimum it should have offered to publish a note acknowledging the failed replication.
A failed replication doesn't necessarily mean anything beyond that the second team failed to replicate the experiment. It doesn't disprove the initial study's positive results, however it may hint at a study having validation issues.
(2019-07-03, 06:40 AM)letseat Wrote: [ -> ]A failed replication doesn't necessarily mean anything beyond that the second team failed to replicate the experiment. It doesn't disprove the initial study's positive results, however it may hint at a study having validation issues.
I'm no scientist but it was my understanding that if something is failed to be reproduced in an experiment, then the first study is put into doubt, no?
(2019-07-03, 06:40 AM)letseat Wrote: [ -> ]A failed replication doesn't necessarily mean anything beyond that the second team failed to replicate the experiment. It doesn't disprove the initial study's positive results, however it may hint at a study having validation issues.
In this case they reported three replications, once of which was a direct replication whose statistical power - the probability of finding an effect of the size previously reported, if it's there - was 80%. So they'd have had to be quite unlucky to miss it. The draft paper can be found here:
https://osf.io/86xhp/
(2019-07-03, 06:47 AM)diverdown Wrote: [ -> ]I'm no scientist but it was my understanding that if something is failed to be reproduced in an experiment, then the first study is put into doubt, no?
The first experiment should never be taken at face value as something consistent. An interesting find without replication is just that. We used to know this, but science journalism -especially in mainstream publications- has made the headline its real star... And there is nothing glamorous about retraction.
(2019-07-03, 06:47 AM)diverdown Wrote: [ -> ]I'm no scientist but it was my understanding that if something is failed to be reproduced in an experiment, then the first study is put into doubt, no?
I suppose I could have been more clear. Yes it produces doubt, but no it is not a disproof. To disprove it they would need to find the methodological flaws within the study that would have lead to a positive result where replications failed to find that same positive result.
(2019-07-03, 07:23 AM)Chris Wrote: [ -> ]
In this case they reported three replications, once of which was a direct replication whose statistical power - the probability of finding an effect of the size previously reported, if it's there - was 80%. So they'd have had to be quite unlucky to miss it. The draft paper can be found here:
https://osf.io/86xhp/
I would certainly say three failed replications casts three times as much doubt upon the original study. The experiment itself is a simple test as well, but I wonder how the original experiment went wrong.
The whole replication crisis certainly gives me misgivings about the field of science as a whole, that major journals rarely have the ethical fortitude to publish negative results let alone fund replications is quite disturbing.