Someone kindly let me know about this paper published in PeerJ early last year by Richard Wiseman, Caroline Watt and Diana Kornbrot, entitled "Registered reports: An early example and analysis."
https://peerj.com/articles/6232/
Here is the abstract:
"The recent ‘replication crisis’ in psychology has focused attention on ways of increasing methodological rigor within the behavioral sciences. Part of this work has involved promoting ‘Registered Reports’, wherein journals peer review papers prior to data collection and publication. Although this approach is usually seen as a relatively recent development, we note that a prototype of this publishing model was initiated in the mid-1970s by parapsychologist Martin Johnson in the European Journal of Parapsychology (EJP). A retrospective and observational comparison of Registered and non-Registered Reports published in the EJP during a seventeen-year period provides circumstantial evidence to suggest that the approach helped to reduce questionable research practices. This paper aims both to bring Johnson’s pioneering work to a wider audience, and to investigate the positive role that Registered Reports may play in helping to promote higher methodological and statistical standards."
The key finding is that of 131 hypotheses for which results were reported in 25 registered reports, 11 (8.4%) were significant [at 5%], compared with 66 out of 232 in 60 non-registered reports (28.4%). The authors conclude that this is consistent with the notion that the use of registered reports reduced the incidence of questionable research practices, and so reduced the percentage of significant results.
They tested for the alternative possibilities that the reduction represented a difference between the two groups in the type of psi being studied (ESP versus PK) or improvements in methodology over time, and found no evidence for them. However, they acknowledge that there are other possibilities they didn't test for. I'd say the most important one is the possibility that there was a difference in the average power of studies in the two groups. If the average power (i.e. the probability of finding an effect if it exists) were lower for the registered reports, one would expect a smaller percentage of significant results. But of course, selective publication of the non-registered studies is a plausible explanation for at least part of the difference between the two groups.
It's worth noting that although the success rate in the registered reports was lower, overall it almost achieves statistical significance at 5% (p=0.064). On this, the authors comment that "this result is not compelling evidence for the existence of psychic ability as the experiments may have contained other non-obvious methodological shortcomings, such as issues regarding sensory leakage and poor randomisation."
https://peerj.com/articles/6232/
Here is the abstract:
"The recent ‘replication crisis’ in psychology has focused attention on ways of increasing methodological rigor within the behavioral sciences. Part of this work has involved promoting ‘Registered Reports’, wherein journals peer review papers prior to data collection and publication. Although this approach is usually seen as a relatively recent development, we note that a prototype of this publishing model was initiated in the mid-1970s by parapsychologist Martin Johnson in the European Journal of Parapsychology (EJP). A retrospective and observational comparison of Registered and non-Registered Reports published in the EJP during a seventeen-year period provides circumstantial evidence to suggest that the approach helped to reduce questionable research practices. This paper aims both to bring Johnson’s pioneering work to a wider audience, and to investigate the positive role that Registered Reports may play in helping to promote higher methodological and statistical standards."
The key finding is that of 131 hypotheses for which results were reported in 25 registered reports, 11 (8.4%) were significant [at 5%], compared with 66 out of 232 in 60 non-registered reports (28.4%). The authors conclude that this is consistent with the notion that the use of registered reports reduced the incidence of questionable research practices, and so reduced the percentage of significant results.
They tested for the alternative possibilities that the reduction represented a difference between the two groups in the type of psi being studied (ESP versus PK) or improvements in methodology over time, and found no evidence for them. However, they acknowledge that there are other possibilities they didn't test for. I'd say the most important one is the possibility that there was a difference in the average power of studies in the two groups. If the average power (i.e. the probability of finding an effect if it exists) were lower for the registered reports, one would expect a smaller percentage of significant results. But of course, selective publication of the non-registered studies is a plausible explanation for at least part of the difference between the two groups.
It's worth noting that although the success rate in the registered reports was lower, overall it almost achieves statistical significance at 5% (p=0.064). On this, the authors comment that "this result is not compelling evidence for the existence of psychic ability as the experiments may have contained other non-obvious methodological shortcomings, such as issues regarding sensory leakage and poor randomisation."
![[-]](https://psiencequest.net/forums/images/collapse.png)
