Wiseman et al analysis of registered reports in the EJP

3 Replies, 846 Views

Someone kindly let me know about this paper published in PeerJ early last year by Richard Wiseman, Caroline Watt and Diana Kornbrot, entitled "Registered reports: An early example and analysis."
https://peerj.com/articles/6232/

Here is the abstract:
"The recent ‘replication crisis’ in psychology has focused attention on ways of increasing methodological rigor within the behavioral sciences. Part of this work has involved promoting ‘Registered Reports’, wherein journals peer review papers prior to data collection and publication. Although this approach is usually seen as a relatively recent development, we note that a prototype of this publishing model was initiated in the mid-1970s by parapsychologist Martin Johnson in the European Journal of Parapsychology (EJP). A retrospective and observational comparison of Registered and non-Registered Reports published in the EJP during a seventeen-year period provides circumstantial evidence to suggest that the approach helped to reduce questionable research practices. This paper aims both to bring Johnson’s pioneering work to a wider audience, and to investigate the positive role that Registered Reports may play in helping to promote higher methodological and statistical standards."

The key finding is that of 131 hypotheses for which results were reported in 25 registered reports, 11 (8.4%) were significant [at 5%], compared with 66 out of 232 in 60 non-registered reports (28.4%). The authors conclude that this is consistent with the notion that the use of registered reports reduced the incidence of questionable research practices, and so reduced the percentage of significant results.

They tested for the alternative possibilities that the reduction represented a difference between the two groups in the type of psi being studied (ESP versus PK) or improvements in methodology over time, and found no evidence for them. However, they acknowledge that there are other possibilities they didn't test for. I'd say the most important one is the possibility that there was a difference in the average power of studies in the two groups. If the average power (i.e. the probability of finding an effect if it exists) were lower for the registered reports, one would expect a smaller percentage of significant results. But of course, selective publication of the non-registered studies is a plausible explanation for at least part of the difference between the two groups.

It's worth noting that although the success rate in the registered reports was lower, overall it almost achieves statistical significance at 5% (p=0.064). On this, the authors comment that "this result is not compelling evidence for the existence of psychic ability as the experiments may have contained other non-obvious methodological shortcomings, such as issues regarding sensory leakage and poor randomisation."
[-] The following 1 user Likes Guest's post:
  • Laird
As I understand it they have proven a file drawer effect is psi studies. It’s however not only in psi reproducibility is a major issue. Just think medicine.....
(2020-01-06, 01:50 PM)sbu Wrote: As I understand it they have proven a file drawer effect is psi studies. It’s however not only in psi reproducibility is a major issue. Just think medicine.....

They are a bit more cautious than that, as they follow their statement that the results are consistent with registered reports reducing the incidence of QRPs, with the following caveat:
"However, the EJP studies were not randomly allocated to condition, so the RRs and non-RRs may have varied on several other factors (including, for example, study design, power, and methodological quality), therefore it is possible that these factors may be responsible for the observed effect. As a result, the findings should be seen as circumstantial, rather than definitive, evidence for the notion that RRs help prevent QRPs."

But personally I'd have been surprised if they hadn't found any indication of selective publication (or other questionable research practices), based on anecdotal evidence and plain human nature. It's obviously a potential problem - though not just in parapsychology, as you point out.
This was officially published on 16 January, as highlighted on the SPR Facebok page.

  • View a Printable Version
Forum Jump:


Users browsing this thread: 1 Guest(s)