Dean Radin's 1989 Neural Networks paper

Ah, thanks. I hadn't read the paper and probably should have indicated that - I had simply assumed sampling without replacement. But anyhow, based on the formula supplied on this page, your conclusion wouldn't change at all even if the sampling had been without replacement; only the probabilities would have changed. That is, it would still have taken 4 or more correct matches to have achieved significance at the 0.05 level, it's just that the probabilities would have been, for 3 or more matches, 0.080 (i.e., for sampling without replacement) where the binomial probability you cited is 0.077 (for sampling with replacement), and, for 4 or more matches (without replacement), 0.019, where the binomial probability you cited (with replacement) is 0.017.

[*]I didn't check the derivation of the formula, but I did confirm that it gives the right results for n=4.

So, your criticism seems sound. That said, it seems unfortunate that parapsychologists like Dean cop it no matter what: if, as in the Ganzfeld, they compare with theoretical expectation, then they are criticised for not comparing with controls; if, though, as in studies like this, they compare with controls, then they are criticised for not comparing with theoretical expectation! Probably best would be to do both, but even then, can we be sure that they would not still be criticised...?...

[*]

In this case theoretical expectation is giving only a rough idea, because the random element of the neural network and the averaging complicate things. So I think comparison with controls is the way to do it properly, but the comparison Radin did wasn't valid.

[Laird]

In this case theoretical expectation is giving only a rough idea, because the random element of the neural network and the averaging complicate things. So I think comparison with controls is the way to do it properly, but the comparison Radin did wasn't valid.

OK, thanks for clarifying. So, just to check that I'm understanding correctly:

Your criticism is not that the comparison should have been with theoretical expectation. You only included the calculation of theoretical expectation as a sort of sanity check. Your criticism instead is that Dean should have run a large number of controls rather than just one, and then determined how much of an outlier his actual result was within the distribution of control data in order to determine a p-value.

(Apologies if that wasn't expressed as precisely as it ought to be, and please feel free to correct my wording).

OK, thanks for clarifying. So, just to check that I'm understanding correctly:

Your criticism is not that the comparison should have been with theoretical expectation. You only included the calculation of theoretical expectation as a sort of sanity check. Your criticism instead is that Dean should have run a large number of controls rather than just one, and then determined how much of an outlier his actual result was within the distribution of control data in order to determine a p-value.

(Apologies if that wasn't expressed as precisely as it ought to be, and please feel free to correct my wording).

Exactly. The theoretical calculation gives us a rough but strong indication that the results aren't statistically significant.

[Laird]

Exactly. The theoretical calculation gives us a rough but strong indication that the results aren't statistically significant.

OK. Your criticism seems to me to be valid, albeit that I am something of a statistics novice.

I'm a few pages into the paper - if I finish it I might post a follow-up.

Thanks for sharing your thoughts!