Transparent Psi Project

Courtesy of the SPR Facebook page, here are details at the Open Science Framework of a huge (1700-5600 participants) attempt to replicate Experiment number 1 from Daryl Bem's "Feeling the Future" paper (precognitive detection of erotic stimuli). The project is funded by the Bial Foundation. I don't recognise the names of any of the people involved in the project:
https://osf.io/72dzf/

(Edit: Here's a better link - to the home page of the project: https://osf.io/jk2zf/ )

Courtesy of the SPR Facebook page, here are details at the Open Science Framework of a huge (1700-5600 participants) attempt to replicate Experiment number 1 from Daryl Bem's "Feeling the Future" paper (precognitive detection of erotic stimuli). The project is funded by the Bial Foundation. I don't recognise the names of any of the people involved in the project:
https://osf.io/72dzf/

Searching for more information, I came across a blog post by Andrew Gelman from about four weeks ago, intended to explain why he is refusing to participate in the project:
http://andrewgelman.com/2017/06/11/im-no...i-project/

Gelman insists there is a problem with "small effect sizes and poor measurements" in this protocol. He doesn't explain what he means by "poor measurements", and I can't really imagine what it could be, given that it's simply a question of determining whether participants can guess the random position of a target more often than chance. He goes on to say "there’s a 5% chance you’ll see something statistically significant just by chance", only for the person inviting him to take part to point out patiently that the experimental design is based on an estimate of "a p = 0.0004 false positive chance". 

Gelman also points out an example of what he thinks "serious work" in the area of psi would amount to:
http://marginalrevolution.com/marginalre...ernet.html
Though he does add that "I’ve not actually read the paper linked there so this may be a bad example." Presumably he didn't even bother to read the web page he cited, let alone the paper it linked to, because although it has "Telepathy" in the title, it then goes on to explain that it was a question of one person's brain being stimulated based on EEG measurements of another person's brain. 

To my mind, the remarkable thing is that Gelman posted a summary of this correspondence on his blog, without any apparent embarrassment (though his correspondent didn't point out to him the blunder over the "Telepathy" paper, so he may still be unaware of it). 

To what extent does this affect what was quoted from Gelgel in the "Statistical significance" thread from 2017? I recall Linda and Steve going 'all in' backing his stance and now he is admitting to being selective about the literature.

To what extent does this affect what was quoted from Gelgel in the "Statistical significance" thread from 2017? I recall Linda and Steve going 'all in' backing his stance and now he is admitting to being selective about the literature.

I can't make any sense of that blog article. Gelman has commented rather a lot about Bem's work in the past, but now he says "I’m not familiar with the ESP experiment".

However, he then goes on to say:
I still think it’s hopeless. The problem (which I’ll say using generic units as I’m not familiar with the ESP experiment) is: suppose you have a huge sample size and can detect an effect of 0.003 (on some scale) with standard error 0.001. Statistically significant, preregistered, the whole deal. Fine. But then you could very well see an effect of -0.002 with different people, in a different setting. And -0.003 somewhere else. And 0.001 somewhere else. Etc. You’re talking about effects that are indistinguishable given various sources of leakage in the experiment.

Then further on he says " I’m not sure what other types of leakage might exist—it’s my impression that Bem’s experiments had various problems."

What on earth is he talking about? This is a precognition experiment. The "leakage" would have to be information transfer from the future to the past. Even if he's suggesting there's some kind of flaw in Bem's protocol - which I've never heard suggested, despite the fact Bem made the software freely available specifically to encourage replication attempts - it really should be easy to rule out "leakage" in a replication that's designed with any degree of competence. And after all, this blog article represents Gelman's rationale for refusing to participate in the design of the replication!

I had another look at the web pages to try to figure out what's actually going on with this project, but without much success. Some kind of consensus has apparently been reached about the first version of an experimental protocol, and I even saw a list of the people involved at one stage, but I couldn't discover what they'd actually agreed. 

If this is transparent science, I hope I never encounter the opaque kind!

I can't make any sense of that blog article. Gelman has commented rather a lot about Bem's work in the past, but now he says "I’m not familiar with the ESP experiment".

However, he then goes on to say:
I still think it’s hopeless. The problem (which I’ll say using generic units as I’m not familiar with the ESP experiment) is: suppose you have a huge sample size and can detect an effect of 0.003 (on some scale) with standard error 0.001. Statistically significant, preregistered, the whole deal. Fine. But then you could very well see an effect of -0.002 with different people, in a different setting. And -0.003 somewhere else. And 0.001 somewhere else. Etc. You’re talking about effects that are indistinguishable given various sources of leakage in the experiment.

Then further on he says " I’m not sure what other types of leakage might exist—it’s my impression that Bem’s experiments had various problems."

What on earth is he talking about? This is a precognition experiment. The "leakage" would have to be information transfer from the future to the past. Even if he's suggesting there's some kind of flaw in Bem's protocol - which I've never heard suggested, despite the fact Bem made the software freely available specifically to encourage replication attempts - it really should be easy to rule out "leakage" in a replication that's designed with any degree of competence. And after all, this blog article represents Gelman's rationale for refusing to participate in the design of the replication!

Well just another reason I don't respect skeptics. Much like the religious, they're the equivalent of children fighting for parents attention, wanting to turn around and go "Ha Ha, mommy likes me better" because of their submissiveness towards what they've decided is reality. Because apparently you just have to accept whatever's in front of you, you can't take steps to change it into what you want, no one's ever done that successfully before.

Maybe he and all other skeptics should stop being worthless cowards and work to make ESP real rather than complaining that other people are trying to build something that they haven't seen built before?

Well just another reason I don't respect skeptics. Much like the religious, they're the equivalent of children fighting for parents attention, wanting to turn around and go "Ha Ha, mommy likes me better" because of their submissiveness towards what they've decided is reality. Because apparently you just have to accept whatever's in front of you, you can't take steps to change it into what you want, no one's ever done that successfully before.

Maybe he and all other skeptics should stop being worthless cowards and work to make ESP real rather than complaining that other people are trying to build something that they haven't seen built before?

Well, to be fair, the people who did comment on the protocol were about evenly divided between proponents and sceptics - around a dozen of each. I'm afraid I can't now find the list I saw, and the website is such a pain to navigate that I can't be bothered to spend any longer looking.

Well, to be fair, the people who did comment on the protocol were about evenly divided between proponents and sceptics - around a dozen of each. I'm afraid I can't now find the list I saw, and the website is such a pain to navigate that I can't be bothered to spend any longer looking.

Well it's more the arguments that made me stop respecting skeptics. It's one thing to say "Hey I don't think that will work, her's why, here's something else you might be able to try instead." and another to say "That's not going to work, stop trying, you're just wasting your time."

I've only ever heard the latter from skeptics. The guy you quoted is a textbook example of it. No real interest in pushing things forward, just looking to tell everyone to give up because "mommy reality said so". And not lifting a finger to fix any potential problems he claims there are with the experiments.

I can't make any sense of that blog article. Gelman has commented rather a lot about Bem's work in the past, but now he says "I’m not familiar with the ESP experiment".

However, he then goes on to say:
I still think it’s hopeless. The problem (which I’ll say using generic units as I’m not familiar with the ESP experiment) is: suppose you have a huge sample size and can detect an effect of 0.003 (on some scale) with standard error 0.001. Statistically significant, preregistered, the whole deal. Fine. But then you could very well see an effect of -0.002 with different people, in a different setting. And -0.003 somewhere else. And 0.001 somewhere else. Etc. You’re talking about effects that are indistinguishable given various sources of leakage in the experiment.

Then further on he says " I’m not sure what other types of leakage might exist—it’s my impression that Bem’s experiments had various problems."

What on earth is he talking about? This is a precognition experiment. The "leakage" would have to be information transfer from the future to the past. Even if he's suggesting there's some kind of flaw in Bem's protocol - which I've never heard suggested, despite the fact Bem made the software freely available specifically to encourage replication attempts - it really should be easy to rule out "leakage" in a replication that's designed with any degree of competence. And after all, this blog article represents Gelman's rationale for refusing to participate in the design of the replication!

Bizarre. But, this is more than a simple Freudian slip. I get the impression that he worked his arguments backwards, beginning with a conclusion and then trying to justify by elaborating a foundation for it.

Bizarre. But, this is more than a simple Freudian slip. I get the impression that he worked his arguments backwards, beginning with a conclusion and then trying to justify by elaborating a foundation for it.

Gelman's original criticism of Bem's work was that there was scope for multiple hypotheses, as expressed in this paper for example:
http://www.stat.columbia.edu/~gelman/res...acking.pdf

For a few of Bem's experiments I think there was a degree of validity to that criticism, because the experimental design did permit more than one natural hypothesis, though when the available hypotheses are actually counted they seem wholly inadequate for the overall results to be explained in this way.

But in any case, this kind of carefully planned study, where every detail is preregistered, obviously rules out any possibility of multiple hypotheses. If Gelman really believed that was the explanation for Bem's results, I'd have expected him to welcome the project and give it all the assistance he could. Instead, he's now talking about other flaws in Bem's protocol, though he can't say what they are. It does give the impression that he's hedging his bets in case the replication produces positive results. But I think anyone who raises post hoc objections to the protocol, having been given the opportunity to comment beforehand and refused, is going to look a bit silly.