Intellectual humility

A pair of articles by Daniel Engber related to this:

Daryl Bem Proved ESP Is Real
Is Science Broken?

Plenty in here to offend everyone who's replied so far, I'd say  But I think it's a fair perspective.

It's no surprise that he is going to defend his own hyperbole.  However, he does a good job of outlining the details of what happened, and the concerns about parapsychology/psychology's use of questionable research practices (QRP's). It's nice to see that he spent no time on the arguments over Bayes' Factors, and instead focussed on what turned out to be the real problem(s) with Bem's research - e.g. selecting some results for publication from a larger pool of experiments, and hypothesizing after he knew which few of the dozens of potential hypotheses the results could be jiggered to support. And it offered some context for why Bem continued to make false statements about his research practices (like claiming that there were no other unreported pilot studies, or that his hypotheses were pre-set). 

What's interesting is that Engber completely missed out that the same process had already run through the field of medicine starting 30 years ago (he makes the mistake of suggesting that this replication crisis may be poised to run through other fields, including medicine). In that case, there wasn't a paper about to be published which showed impossible results, but a growing realization that the sort of information which physicians used to guide their practice wasn't always the best information in terms of evidentiary level. That closer look at the "level of evidence" used to guide practice didn't just show that physicians weren't always using the highest level of evidence. It also laid bare where evidence was lacking for practices which had a long, seemingly solid foundation. That focus on "level of evidence" changed research practices in ways that address the concerns Engber mentions, by prioritizing research performed at the highest level of evidence appropriate to the type of research (for example, the factors which come into play are different between pre-clinical studies and post-marketing studies). It should be noted that "level of evidence" is essentially synonymous with "replicability". That is, the factors which lead to "strong" evidence are the same factors which lead to "replicability" - they are the factors which make the hypothesis more likely to be true/valid.

This is why the potential problems with Bem's research were obvious to me from the beginning. I already had decades of practice (in another field which depended upon social science methods for some of its research (i.e. medicine)) identifying those QRPs and understanding how they would produce false results from seemingly solid methodology. 

Linda

Regarding the claim in the first of those articles that Bem did unreported pilot testing, it's worth noting his comment about that to Ulrich Schimmack, quoted here:
https://psiencequest.net/forums/thread-c...4#pid13704

I know there's a lot of back and forth in that thread, in addition to all the Bem stuff on Skeptiko, but I'm curious as to your overall take away.

Does Bem show there's precognition or not? Or is the answer "maybe"?

I know there's a lot of back and forth in that thread, in addition to all the Bem stuff on Skeptiko, but I'm curious as to your overall take away.

Does Bem show there's precognition or not? Or is the answer "maybe"?

I hope to do some more analysis of the data when time permits. But based on what I've done so far:
(1) Of course if people think that Bem did a lot more studies and suppressed the unsuccessful ones, that would be a possible explanation. He says that's not the case.
(2) In some cases the experimental design isn't just geared towards a simple comparison of two groups of trials of equal size, but contains more than two groups of trials and therefore permits additional hypotheses to be tested. It's obvious from the data files where this happens. But as far as I can see so far, this isn't sufficient to explain the results.

I hope to do some more analysis of the data when time permits. But based on what I've done so far:
(1) Of course if people think that Bem did a lot more studies and suppressed the unsuccessful ones, that would be a possible explanation. He says that's not the case.
(2) In some cases the experimental design isn't just geared towards a simple comparison of two groups of trials of equal size, but contains more than two groups of trials and therefore permits additional hypotheses to be tested. It's obvious from the data files where this happens. But as far as I can see so far, this isn't sufficient to explain the results.

Thanks for that. Another question - given what feels like inevitable controversy whenever research of this sort arises, what do you think of Braude's commentary on quantitative Psi research:

In my view, what’s always been needed are, first, psi effects so impres-sive that quantitative analyses are beside the point. These effects are plentiful enough, and include some ganzfeld and remote viewing hits that are so spot-on or so reliable that it’s simply absurd to attribute the successes to chance. They also include PK results so dramatic and obtained under conditions so obviously clean that allegations of fraud are clearly and merely lame cries of protest (see Braude, 1997, Braude, 2007). And second, what’s needed are gifted subjects who can produce results relatively consistently, with different experimenters and on numerous occasions. This helps diffuse the notorious “source of psi”  problem, which arises acutely when conducting tests with unselected subjects. In those cases, there are so many unidentifiable and uncontrollable variables in the underlying causal nexus that it’s never clear why the experimental results turned out as they did (see my Editorial in JSE 23:3, Fall 2009). These are cases for which ineffective quantitative analysis is the only way of defending the claim that something paranormal has occurred. But if you have a star subject who is regularly associated with conspicuously anomalous effects, we have at least a prima facie case for assigning that subject a key role in the causal nexus, and quantitative analysis quickly becomes irrelevant. For example, Joe McMoneagle’s remote-viewing track record speaks for itself, irrespective of its statistical improbability, and no matter what other psychic in
fluences might have partially contributed (positively or negatively) to the observed result.

Admittedly it's been some time since I looked at the McMoneagle stuff, so cannot comment on that example much, but I kinda see Braude's point that a lot of effort is wasted on trying to suss out potential Psi ability in the general population. Of course how to find the Psi superstars, or at least more gifted, is another question altogether...

p.s. A separate thread for Braude's editorial is here

Thanks for that. Another question - given what feels like inevitable controversy whenever research of this sort arises, what do you think of Braude's commentary on quantitative Psi research:


Admittedly it's been some time since I looked at the McMoneagle stuff, so cannot comment on that example much, but I kinda see Braude's point that a lot of effort is wasted on trying to suss out potential Psi ability in the general population. Of course how to find the Psi superstars, or at least more gifted, is another question altogether...

p.s. A separate thread for Braude's editorial is here

I think the problem is if you rely on - for example - striking results for individual remote viewing sessions, then that just shifts the debate to a discussion of how many sessions you would need to produce a small number of such good quality by chance, and how that compares to the number of sessions that have been done. In response to a small number of extremely impressive examples, sceptics will invoke the law of truly large numbers, with some justification.

I'd have thought from an evidential point of view, the best of both worlds would be to carry out quantitative experiments with star performers (if they can really be found). Of course, a drawback of that approach would be that replication would become even harder if it depended on finding a star performer, and that we wouldn't learn much about psi in the population as a whole.

I think the problem is if you rely on - for example - striking results for individual remote viewing sessions, then that just shifts the debate to a discussion of how many sessions you would need to produce a small number of such good quality by chance, and how that compares to the number of sessions that have been done. In response to a small number of extremely impressive examples, sceptics will invoke the law of truly large numbers, with some justification.

I'd have thought from an evidential point of view, the best of both worlds would be to carry out quantitative experiments with star performers (if they can really be found). Of course, a drawback of that approach would be that replication would become even harder if it depended on finding a star performer, and that we wouldn't learn much about psi in the population as a whole.

Another thing to consider is can we take the average person and make them "gifted" with regards to Psi ability. I agree with you that quantitative, reproducible results with Psi-stars is the best of both sides of the argument, but the search for those individuals who might have exceptional capabilities definitely would be searching for needles in a haystack.

The only way to cut through the impasse, it seems to me, is to work on finding that which could bolster Psi results. Perhaps that is some dose of psychedelics, perhaps its some ritualized practice, etc. This could also, in turn, tell us something about Psi in the general population.

Also, I'm curious as to what results you think have told us anything about Psi in the general population?

Another thing to consider is can we take the average person and make them "gifted" with regards to Psi ability. I agree with you that quantitative, reproducible results with Psi-stars is the best of both sides of the argument, but the search for those individuals who might have exceptional capabilities definitely would be searching for needles in a haystack.

The only way to cut through the impasse, it seems to me, is to work on finding that which could bolster Psi results. Perhaps that is some dose of psychedelics, perhaps its some ritualized practice, etc. This could also, in turn, tell us something about Psi in the general population.

Also, I'm curious as to what results you think have told us anything about Psi in the general population?

I was thinking more of what we might learn in the future.

I was thinking more of what we might learn in the future.

I guess the challenge as I see it revolves around finding good enough results that they are largely unassailable, in the hopes that this provides enough funding to the field to learn something about the general population's capacity for Psi.

Thus results like Bem's seem destined to sit in a mire of endless debate rather than moving the needle?

I guess the challenge as I see it revolves around finding good enough results that they are largely unassailable, in the hopes that this provides enough funding to the field to learn something about the general population's capacity for Psi.

Thus results like Bem's seem destined to sit in a mire of endless debate rather than moving the needle?

Yes, in that case I agree. I don't think progress will be made without a lot more resources - human and material. If clear results could be produced under conditions that were unassailable, then no doubt that would help a lot, even if replicability were still a problem.

While I don't think a plausible non-psi explanation has been offered for Bem's results, I think the real problem there is that replicability has again been only hit-and-miss.