(Yesterday, 09:36 AM)Max_B Wrote: If further studies don't replicate Bem's erotic image experiment, they are not direct replications, they are testing some other assumptions. The attempted replication study sbu linked to above was conducted online (not in a laboratory); participants were recruited by some online recruitment service with an average age of 50 years old (not Cornell undergraduates).
Bem's experiment went and grabbed modern erotic images off porn sites that appealed to young males. Cornell reports undergrads over 25 as comprising only 2% of their population, 98% are under 25 usually between ~18-22 y/o. Bem was very specific - he found the young males that did best were high stimulus seekers. Then you've got the experimenter effect, even Dr Carolyn Watt accepts that this effect exists - an online experiment excludes that effect.
sbu also linked to another failed replication some weeks ago, when I looked at that study... they ran the experiment at the same time with all the male and female subjects together in the same room; the recruitment population pool was different from Bems and age was much higher, they also couldn't use the highly erotic images Bem used (no explanation why).
Note that in this meta-analysis by Bem from 2016 https://pmc.ncbi.nlm.nih.gov/articles/PMC4706048/ his standard for an exact replication was simply using his original software:
Quote: For this last variable, each experiment was categorized into one of three categories: an exact replication of one of Bem’s experiments (31 experiments), a modified replication (38 experiments), or an independently designed experiment that assessed the ability to anticipate randomly-selected future events in some alternative way (11 experiments). To qualify as an exact replication, the experiment had to use Bem’s software without any procedural modifications other than translating on-screen instructions and stimulus words into a language other than English if needed.
So the argument that TPP wasn't a proper replication because of population or setting differences doesn't hold up by Bem's own criteria. TPP tested his core hypothesis, and did so under proper scientific conditions that his original work lacked.
What made TPP different: protocols were pre-registered to prevent cherry-picking results. Multiple independent labs ran identical tests to rule out lab-specific effects. Strict blinding prevented experimenter influence. All data, code, and deviations were made public.
These safeguards matter because Bem's original studies didn't have them. Different samples and contexts can affect results, but methodological weaknesses are the more likely explanation for positive findings. When you control for these issues like TPP did, the effects disappear.
Mastodon