Call for retraction of "Feeling the Future"

Yep, it's in an update right up the top of the original blog post calling for retraction.

Thanks. Prof. Schimmack previously circulated a draft of the letter for comment, including to some of those who had commented on his blog. I see that three more people have put their names to the letter - Dr Linda Schultz, Dr Rickard Carlsson and Dr Stefan Schmukle.

My own comments on the first draft of the letter were as follows:

I'm afraid I don't yet feel I understand what's going on with these experiments, and I think I will need quite a long and detailed look at the data before I stand a chance of doing that. 

I do think from what we know already that it would be reasonable for the author to correct the description of Experiment 5 that says there were no positive images, and also to clarify that the design of that experiment, and also that of Experiment 1, allowed for the testing of additional hypotheses that turned out not to produce significant results. I think those additional hypotheses should have beeen declared in the file drawer section, or preferably in the main results sections. (However, I do find it difficult to believe that Bem intended the description of Experiment 5 to deceive the reader, considering that he had just quoted the instructions to participants, which referred to a range of images including "very pleasant" ones. Obviously it's unfortunate that the referees didn't pick up on the contradiction there, and the obvious availability of alternative hypotheses in Experiment 1 as described in the paper.)

My other thought is that, regarding the statistical peculiarities that you number 1-3, and the decline effect you have found, they obviously show that the data don't reflect a well behaved phenomenon in which the effect size remains steady and individual trials are statistically independent.  The trouble is that where psi is concerned it's not clear whether that is enough to conclude that the data are spurious, because we don't know what the statistical properties of psi should be. We know how the statistics should behave in the absence of psi, so testing the null hypothesis is a rational way of proceeding, but doing the converse - assuming psi exists and testing whether the data are consistent with that assumption - doesn't necessarily make sense. I realise that's the kind of argument sceptics find infuriating, but if we allow for the possibility of things like psi experimenter effects, then psi really may not be statistically well behaved at all.

I think if it were possible to come up with a definite "recipe" by which the observed decline effect and the statistical properties 1-3 could be generated from chance data, then your case would become much stronger - though after looking at the dates in the data file I was left with the impression that there wasn't really time to fit in the kind of numbers of unreported participants that would be needed in the scenario you suggested. 

If the letter had been in its present form, I'd also have commented that - while Schimmack's scenario of discarding pilot experiments if unsuccessful and continuing them if successful would, in qualitative terms, produce a decline effect - I'm not convinced it would produce a decline effect which quantitatively matched the observed one.

I see that three more people have put their names to the letter - Dr Linda Schultz, Dr Rickard Carlsson and Dr Stefan Schmukle.

Rickard Carlsson is a senior lecturer in the Department of Psychology at Linnaeus University in Sweden, and Stefan Schmukle is a professor in the Department of Psychology at the University of Leipzig.

[Laird]

Thanks. Prof. Schimmack previously circulated a draft of the letter for comment, including to some of those who had commented on his blog. I see that three more people have put their names to the letter - Dr Linda Schultz, Dr Rickard Carlsson and Dr Stefan Schmukle.

My own comments on the first draft of the letter were as follows:

[elided]

Thanks for sharing that, Chris. It all seems pretty reasonable to me. Not being particularly statistically literate, it is hard for me to contribute to this conversation. Lacking sufficient statistical literacy, I can only take at face value the claims of various skeptical folk that, e.g., the data lack plausibility because they are "too good to be true" given the supposed effect size. On the other hand, if Daryl Bem was fudging the data in the way he would need to have to have achieved these "too good to be true" results, then I think that that which you contribute here is key:

I think if it were possible to come up with a definite "recipe" by which the observed decline effect and the statistical properties 1-3 could be generated from chance data, then your case would become much stronger - though after looking at the dates in the data file I was left with the impression that there wasn't really time to fit in the kind of numbers of unreported participants that would be needed in the scenario you suggested. 

If the letter had been in its present form, I'd also have commented that - while Schimmack's scenario of discarding pilot experiments if unsuccessful and continuing them if successful would, in qualitative terms, produce a decline effect - I'm not convinced it would produce a decline effect which quantitatively matched the observed one.

I agree that (hopefully paraphrasing you accurately) if these results are to be definitively accepted to be a result of - even inadvertent - biased choices such that the paper deserves to be retracted, then a plausible model (i.e. with sufficient detail) needs to be provided as to how this occurred.

Tbf, the replication rate right now for statistics significance is around 10%.

I don't know how interested people are, but the discussion is continuing. 

Linda has compared the statistics of Bem's experiment 101 (published in 2003) and of the first 50 participants in his experiment 5 (published in 2011), and has concluded that - as suspected previously - they are the same. (There is a factor of 10 difference in one statistic, but I assume that's a typo.) 

Because 101 involved 6 classes of images, but the description of 5 made it sound as though there was only one class of images plus controls, Linda says "I think this may be sufficient evidence to consider calling for a retraction." 
https://replicationindex.wordpress.com/2...mment-3486

I'd like to ask Linda (assuming she is the same Linda who posts here as fls) to clarify one of her criticisms of Daryl Bem's paper. I did ask this previously on Prof. Schimmack's blog, but got no response. Possibly my request was missed.

Linda wrote:
Experiment 7
The description of this experiment is different from the initial report, which included strongly negative and erotic pictures. Either Bem neglected to include the results from 146 of the subjects, or neglected to include all the trials from each subject.

Linda provided a link to what she referred to as the "original report" of Experiments 5 and 6, namely a modified version of a presentation given at the 46th Annual Convention of the Parapsychlogical Association in 2003, entitled "Precognitive Habituation: Replicable Evidence for a Process of Anomalous Cognition":
https://pdfs.semanticscholar.org/8033/f0...0b2e73.pdf

But this can't be the initial report of Experiment 7, as the data file shows that this experiment was conducted 18 months after the presentation. I wonder if Linda was referring to a different report of Experiment 7, or if she was mistaken in thinking that Experiment 7 was included in the 2003 report.

I agree. Now that the data (with dates) from Feeling the Future has been made available, Bem does not appear to have included any of the supraliminal experiments (including his experiment on the retroactive induction of boredom) from 2003 in Feeling the Future.

Linda

I agree. Now that the data (with dates) from Feeling the Future has been made available, Bem does not appear to have included any of the supraliminal experiments (including his experiment on the retroactive induction of boredom) from 2003 in Feeling the Future.

Thank you for clarifying that. 

The supraliminal experiment published in 2003 is apparently the one mentioned in the "file drawer" section of "Feeling the Future".

There are a couple more blog posts by Prof. Schimmack mentioning Bem's work:
https://replicationindex.wordpress.com/2...plication/
https://replicationindex.wordpress.com/2...n-studies/

I can't see any more analysis of Bem's results there, though. Rather, Schimmack seems now to take it as read that Bem's results are bogus, and almost seems to use Bem's name as a weapon against those who are more cautious.

I just thought I'd post a link here to Larry Dossey's 2015 article in Explore, entitled "Why Are Scientists Afraid of Daryl Bem?"
http://www.explorejournal.com/article/S1...1/fulltext

It's really about the (scientific) politics of the controversy over Bem's work, and not about the details of the work itself. As one would expect, it makes no attempt to be balanced, but sides wholeheartedly with Bem against his critics. But it does have a useful list of 91 references, which includes contributions on both sides of the argument, as well as material related to the general debate between proponents and sceptics.

Also, I see that there is a new article on Dr Schimmack's blog, presenting a revised version (with parenthetical comments) of a paper he published in 2012, criticising "Feeling the Future" because he would have expected more non-significant results on the basis of the reported effect sizes:
https://replicationindex.wordpress.com/2...gic-index/

At the start, he comments:
"Aside from Alcock’s demonstration of a nearly perfect negative correlation between effect sizes and sample sizes and my demonstration of insufficient variance in Bem’s p-values, Francis’s article and my article remain the only article that question the validity of Bem’s origina findings. Other articles have shown that the results cannot be replicated, but I showed that the original results were already too good to be true. This blog post explains, how I did it."

For comparison, the version of Schimmack's paper published in 2012 is here:
http://citeseerx.ist.psu.edu/viewdoc/dow...1&type=pdf

Alcock's comment about the negative correlation between effect size and sample size is in an online article here:
https://www.csicop.org/specialarticles/s...the_future

And Francis's paper is here:
https://link.springer.com/article/10.375...012-0227-9