Call for retraction of "Feeling the Future"

Can someone explain how pilot study shopping? Can effect results without fudging the data?

The idea is that the selected pilot trials would be included in the final results, together with (non-selected) continuation trials. So, depending on the size of the hypothetical pilot studies, some percentage of the final results would be selected for success, which would bias the overall result in the same direction.

I can see why the scenario of selected pilot studies would have occurred to Schimmack when he found a strong decline effect in the data. The question is whether the data actually support the idea when they are examined in detail. I don't think they do.

I think it's worth repeating that Bem explicitly addressed the issue of unreported trials in his 2011 paper, and referred only to one small study (which had been reported in a previous publication). The quotations attributed to him in the Slate article do seem to suggest otherwise, though.

I can’t say I’m hugely surprised by all of this. Notwithstanding the knee-jerk rejection from certain scientists who instantly reject any evidence in favour of psi, Bem has written material that effectively hands his opponents enough rope to hang him with.

There’s an article on his own website where he writes about how to get scientific papers published

“... at least become intimately familiar with the record of their [the subjects of research] behavior: the data. Examine them from every angle. Analyze the sexes separately. Make up new composite indexes. If a datum suggests a new hypothesis, try to find additional evidence for it elsewhere in the data. If you see dim traces of interesting patterns, try to reorganize the data to bring them into bolder relief. If there are participants you don’t like, or trials, observers, or interviewers who gave you anomalous results, drop them (temporarily). Go on a fishing expedition for something – anything – interesting”

http://dbem.org/WritingArticle.pdf

I believe this is pretty common in science, to look for patterns in the data and then write as if that was what you were looking for all along, but coming from the pen of someone trying to prove something as distasteful as ESP, it’s gold dust to some commentators.

Quite apart from the debate over whether or not psi exists, I don’t think that the author of this piece has done enough to show that Bem’s paper deviates from typical practices in psychology enough to warrent a retraction. At least, not without taking down a lot of other papers with it.

I’ve always though that parapsychology teaches us either that psi exists, or soft sciences really need to radically redefine how they do things.

This isn't the first time that a decline effect has been found when data from psi experiments have been retrospectively analysed. In "The Enchanted Voyager" (1982) his biography of J. B. Rhine, Denis Brian relates what happened in the 1940s when all Rhine's staff except Betty Humphrey were diverted to military service or other war work:

He now thought up a useful occupation for a lab staff of one. For several years, records of PK experiments had piled up and been kept in a basement safe. Here was the opportunity to study them, to see if they might reveal any as yet hidden secret. Rhine set Humphrey the task of reexamining all the results of those experiments. Most particularly, Rhine wanted to know if she would discover a "decline effect" in PK records. ESP subjects invariably scored higher at the start of tests, giving support to the saying "beginners' luck." One explanation for such an effect is that people try harder when they first start out on a task. The falling off in ability is paralleled in memory, learning and sense perception.

Rhine followed Humphrey's analysis like a man watching his horse lead the field. Her final results showed that there was a typical decline curve in PK experiments that appeared when dice were thrown by hand, cup, or machine. To Rhine, the most important aspect of this discover was that no one could have faked those curves, because at the time the tests were undertaken, from 1934 on, no experimenter or subject knew that anyone would later examine the records for that hidden evidence.

With some justification Rhine believed that, as evidence for PK, it was a complete confirmation of the original work. He compared this evidence dredged up from the basement safe to fossils in the strata of the earth. And this material could lead to only one conclusion - that the mind of man could, however slightly, direct rolling dice. PK, in other words, was a fact.
...

Rhine invited independent investigators to check the PK records Humphrey had been through. None accepted, although Gaither Pratt, who could hardly be described as independent, later checked and confirmed Humphrey's findings.

(pp. 170, 171)

(2) In your discussion of selective reporting as a potential explanation for Bem's results, you estimate that this would require 18,000 participants in total (95% of whose results would have to be discarded). This does seem implausible. But have you attempted to calculate corresponding figures for your alternative scenario of running large numbers of pilot studies and continuing only the most promising ones? Obviously this wouldn't require as many participants as running large numbers of full studies, but it seems to me the requirement would be of the same order. Particularly to get an end result of 9 significant studies out of 10, because the promising results of the pilot studies would be diluted by the chance results of the continuations, so a more stringent criterion than 5% significance would need to be applied to the pilots. I wonder whether this explanation is really much more feasible than the other possibilities you reject.

On this point, Schimmack's estimate of 18,000 participants - for the situation in which a large number of full (100 trial) experiments were conducted, and then a small number of successful ones were published - was obtained from the fact that 9 out of 10 of the published experiments were significant at the 5% level (which would be obtained by chance 1 time out of 20), by multiplying 9x20 to get a total of 180 experiments.

There's no right way to do this kind of estimate. We could alternatively say the 9 were also significant at 4%, and get 225, or that 6 were significant at 1.4%, and get 430, or indeed that 4 were significant at 1%, and get 400.

The p values calculated by Schimmack for trials 1-50 of each experiment allow us to make similar estimates for the total number of pilot studies of 50 trials that would be required in his scenario. We could say that 8 out of 10 of the published experiments were significant at 4%, and get 200, or that 5 were significant at 2%, and get 250, or that 3 were significant at 1%, and get 300.

The total number of experiments required is similar in each case, with 95% or more being excluded from publication. In Schimmack's scenario this would require only about half the number of trials, because the abandoned experiments would contain only 50 trials each. But the total number of trials would still be very large - 10,000 or more. 

And on top of that there is the problem that trials 51-100 are still significant at 2%. I don't believe this scenario is viable.

I give up. Who wrote that article?

~~ Paul

I give up. Who wrote that article?

~~ Paul

The article was written by Professor Ulrich Schimmack, a psychologist at the University of Toronto Mississauga:
https://www.utm.utoronto.ca/psychology/f...ack-ulrich

As my comments hadn't appeared, I contacted him, and he advised me that there must have been a problem, that he was happy to receive comments, and that I should try again, which I shall do.

The article was written by Professor Ulrich Schimmack, a psychologist at the University of Toronto Mississauga:
https://www.utm.utoronto.ca/psychology/f...ack-ulrich

Please to always include the authors' names and the date on every published article or paper.

~~ Paul

Please to always include the authors' names and the date on every published article or paper.

~~ Paul

Well, the date was visible on the blog post I linked to. But the author's name was not immediately apparent. However, I did give it when I worked out what it was - in my third post in this thread:
http://psiencequest.net/forums/thread-ca...0#pid13130

Thank you, chris

I’ve always though that parapsychology teaches us either that psi exists, or soft sciences really need to radically redefine how they do things.

Well, which do you have your money on, lyace? You've been scouting out around these forums for a long time, and I do see you as a proponent of some kind who is also fairly skeptical. You don't really think that parapsychology has a good chance of just being fraud? 

I know you are making a point and technically there is always a possibility of something, but one out of context quote from Bem isn't too significant to me if it is anybody else. When the skeptics try and fudge things up, they have always in the past been found out pretty quickly. I've seen no convincing evidence that, especially amongst the more respected and serious parapsychologists, are just tweaking the data to get the results they wanted. With all the dozens of ways information that such fraudulent activity on that grand of scale could get out easily, I don't find it probable at all. 

I mean, there are frauds everywhere, but is parapsychology all it? I can't for the life of me agree to that. Not saying you do, but the fraud card is useless to me at this point.