(2019-01-24, 12:33 AM)Laird Wrote: [ -> ]So, basically, you're unable to justify your contention that selection bias can explain the results, but that's not going to stop you from repeating it ad nauseam. It's probably best to leave it at that then.
I’m open to unconscious bias in a motivated experimenter. The evidence that such phenomena exist is very high. (I have made some justifications, but they’re normally in the bits you decide not to quote in your responses.)
I’m open to the possibility of some experimenter psi, but that wasn’t the hypothesis that was put forward to be falsified here. The evidence that such a phenomenon exists is much harder to determine.
Selected samples are those which are not representative of the underlying population from which they are drawn - in this case, the population of global events capturing our global consciousness. Regardless of whether you favor anomalous or non-anomalous causes, it appears that the events listed as formal hypotheses represent a selected sample.
Selection bias is the effect which is introduced into that sample by the selection process, for example the bias of a motivated experimenter.
The
wikipedia article on this is pretty good at explaining the different kinds of selection and their effects/bias, although it takes more of a medical perspective.
This is what Bancel said about selection wrt the GCP in the 2014 paper:
"Prominent among these [alternate hypotheses] are selection hypotheses, which posit that, for each replication, the experimenter's choice of events and subsequent designation of event time-periods may lead to small biases in the event Z-scores. In these scenarios, the fortuitous selection of naturally occurring random deviations is responsible for the measured effect, rather than a physical influence which alters the RNGs' behavior. The ambiguity is aggravated somewhat by the conception of the GCP experiment. The formal experiment tests only the event Z-scores and does not propose or test for any underlying mechanism or signature of the effect. Furthermore, the latitude allowed in event choices is consistent with selection bias, as this freedom is necessary to generate biased Z-scores."
In his 2016 paper, he goes on to demonstrate that (at least some of) the registered events represent a selected sample.
So even amongst proponents there is general recognition that the conditions are present for selection bias to generate the observed effect, with a selected sample. The remaining question is whether or not the details of the selection bias are anomalous or non-anomalous, given that non-anomalous causes are not prevented (or made even a little bit onerous), and that researchers' use of flexibility often goes unrecognized by themselves (or if recognized is seen as defensible). Statements made about pre-specified statistical tests may be given sincerely, even in the setting of considerable flexibility (as per Gelman and Loken).
(2019-01-24, 05:40 PM)fls Wrote: [ -> ]This is what Bancel said about selection wrt the GCP in the 2014 paper
And here's how he continued a couple of paragraphs down (emboldening mine):
Quote:One can distinguish between conventional and psi models of selection. In the former case, the selection of events and their time-periods is guided by direct information about the data before the replication specifications are determined. Conventional selection can arise from either poor methodology or data-peeking which allows the experimenter to gain prior information about the replication outcome.
You and malf have
still not explained what it is about this experiment's methodology - or, in other words, its blinding procedure - that can account for the results in terms of selection bias. The experimenters are very clear that they were blind: that they pre-specified each event's start and end time, and the method of statistical analysis to be used,
before looking at the data relating to it.
So, either you guys are maintaining that the experimenters are lying, and that they actually peeked at the data, or you are maintaining that this means of blinding is insufficient, or you are maintaining both. In the latter two cases, could either or both of you please
finally explain what is insufficient about this blinding procedure?
(2019-01-25, 12:55 AM)Laird Wrote: [ -> ]And here's how he continued a couple of paragraphs down (emboldening mine):
You and malf have still not explained what it is about this experiment's methodology - or, in other words, its blinding procedure - that can account for the results in terms of selection bias. The experimenters are very clear that they were blind: that they pre-specified each event's start and end time, and the method of statistical analysis to be used, before looking at the data relating to it.
So, either you guys are maintaining that the experimenters are lying, and that they actually peeked at the data, or you are maintaining that this means of blinding is insufficient, or you are maintaining both. In the latter two cases, could either or both of you please finally explain what is insufficient about this blinding procedure?
I don’t know what’s going on, nobody does. We do not have the equivalent of the Hennacy-Powell video. I guess anything Radin is involved in can be judged against his history and we appear to be exploring that in another thread.
The blinding clearly wasn’t sufficient for the proposed hypothesis, such at was. We know this from Bancel's work, which shows they achieved their desired result in the absence of data to support it. Now perhaps their blinding was undone by ‘experimenter psi’, or something else... which brings us back to Radin’s history.
I still don’t understand how events are chosen. Who does that?
A simple "No" would have sufficed, malf. ;-)
(2019-01-25, 12:55 AM)Laird Wrote: [ -> ]So, either you guys are maintaining that the experimenters are lying, and that they actually peeked at the data, or you are maintaining that this means of blinding is insufficient, or you are maintaining both. In the latter two cases, could either or both of you please finally explain what is insufficient about this blinding procedure?
I share your frustration, to the extent that I feel there's a vacuum of critical thinking here and some kind of sensible sceptical criticism of the protocol should be provided.So my attempt to do that is as follows.
By the standards of the 1990s the design of the project was adequate to the extent that it acknowledged the importance of specifying the statistical tests in advance of looking at the data. (And those are tests of the null hypothesis, so their validity isn't dependent on any particular mechanism giving rise to the anomalous behaviour.) The aspect of the design that was flawed - certainly by today's standards, but also in the light of the kind of precautions that used to be taken even as far back as the 1930s and 1940s - was its failure to incorporate any safeguards against questionable research practices, or a mechanism to ensure the statistical tests were unambiguously specified. That's demonstrated by the fact that Bancel found and excluded 13 events where either there was ambiguity in the test or the data had been looked at before the test was specified. So although no one has ever suggested a conventional explanation consistent with the stated protocol, we have no independent assurance that the protocol was observed. That largely depends on trust, which is obviously a great pity. So if anyone were to try to replicate the project, a priority would be automatic procedures to ensure the data remained inaccessible until the statistical tests had been fixed and published. I don't imagine that would be hard to do in practice today.
(2019-01-25, 01:58 AM)malf Wrote: [ -> ]I don’t know what’s going on, nobody does. We do not have the equivalent of the Hennacy-Powell video. I guess anything Radin is involved in can be judged against his history and we appear to be exploring that in another thread.
The blinding clearly wasn’t sufficient for the proposed hypothesis, such at was. We know this from Bancel's work, which shows they achieved their desired result in the absence of data to support it. Now perhaps their blinding was undone by ‘experimenter psi’, or something else... which brings us back to Radin’s history.
I still don’t understand how events are chosen. Who does that?
They make a big point about making the data and its position (z-score) easily accessible - both in real time and historically. So there's no question that people are looking at the data (and encouraged to do so). Since this seems to be at odds with how we are interpreting their stated practice, I suspect there is a discrepancy between what we picture (no data peeking) and what was actually done. I don't think Nelson actually says that the data wasn't sometimes looked at informally, only that the outcome statistic and timestamps were specified prior to close examination, for those events which were analyzed. This is opposed to the early days where Radin would try multiple different time periods, blocking intervals and outcome statistics in order to find one which was statistically significant* (as far as I can tell, Radin has never met a QRP he didn't like). It doesn't mean that some of the researchers aren't looking through the data displays prior to analysis, when wondering about which events may be of interest or trying out various other ideas.
At the time the project was started until the last few years, it was generally unrecognized amongst psychology the extent to which apparently quite reasonable practices were creating effects. For example, Nelson describes that they were thinking of changing the outcome variable from the correlated mean shift to the device variance, in 2002 (on the recipes page). So for a period of time they performed both measures when possible, in order to learn more about the question. It's fairly clear that this sort of flexibility will have introduced bias into the event dataset. Had the device variance happened to not perform well (i.e. produce positive results), we likely would have heard no more about it. But it doesn't get mentioned as a source of flexibility, because it wouldn't have been recognized as such. As you say, who knows what else is in there?
http://noosphere.princeton.edu/data/eggsummary/
https://www.heartmath.org/gci/gcms/live-data/
We are short on details as to how events are chosen. It is clear that there are many people involved in proposing events to be analyzed, without any clear indication of why one is chosen, but another not. The informal perusal of the readily available data would help a researcher decide whether or not an event could be considered "global" (for example).
http://noosphere.princeton.edu/meth_p3.html
http://noosphere.princeton.edu/results.html
ETA: I also forgot about the 100-odd events which end up excluded from the main findings. The distinction for those is vague - "for example, an event may be of great interest in one country, but unknown or little attended in the rest of the world." I don't know how that justification applies to Good Friday, for example.
http://noosphere.princeton.edu/res.informal.html
I think it's a bit silly to propose "experimenter psi" without first addressing this gaping hole in the plan. I am also surprised that proponents are behind this idea, given that other examples where non-anomalous knowledge was available, but less obviously so (such as the Hennacy-Powell video you mentioned) were roundly criticized and dismissed.
Linda
*ETA:
"
There is one exception to this, namely the Y2K analysis Dean Radin did. He said that he tried several variations before choosing the best representation. That case is included in the formal database, with a Bonferonni correction of a factor of 10."
"Radin used a third order statistic on the variance and computed a sliding median (rather than the sliding average) because the above "worked" and variants did not. In addition it only worked with integer time zones leaving significant others out of the 37 zones. Including just them and keeping Dean's exploratory analysis made the "effect" go away."
http://noosphere.princeton.edu/faq.html
Here's a short but quite interesting presentation by Roger Nelson from the Society for Scientific Exploration Conference in 2016 - ten minutes' talk followed by questions. It includes a figure that I don't remember seeing in the published papers, showing an analysis by Peter Bancel concluding that about two thirds of the events showed a positive effect, one sixth no effect, and the other one sixth a "true negative" effect. It would be interesting to see more details of that.
(2019-01-26, 12:25 PM)Chris Wrote: [ -> ]Here's a short but quite interesting presentation by Roger Nelson from the Society for Scientific Exploration Conference in 2016 - ten minutes' talk followed by questions. It includes a figure that I don't remember seeing in the published papers, showing an analysis by Peter Bancel concluding that about two thirds of the events showed a positive effect, one sixth no effect, and the other one sixth a "true negative" effect. It would be interesting to see more details of that.
He mentions Bancel’s findings that solstice events used by the experimenters showed an effect, but the solstice events not included showed no effect. Who chose which solstice events to include? How were those decisions made?
Just a reminder that the Global Consciousness Project has provided a lot of information on its website. This includes a page with details of who originated each hypothesis and links to further information about individual events, often discussing the thinking behind the hypotheses. If people are interested, they can learn more here:
http://noosphere.princeton.edu/results.html
Incidentally, a comment was made earlier which seemed to imply that Dean Radin was heavily involved in the selection of hypotheses early in the project. As far as the Formal Hypothesis Registry is concerned, that is not the case. Radin is noted as an originator for only 4 out of the 513 hypotheses, 3 of them jointly with Nelson and others. None of these 4 events produced a significant result. One of them (number 81) was among the 13 events excluded by Bancel because they weren't prespecified or were ambiguous.
Radin did publish an additional post hoc analysis of the GPC data for 9/11, but that was separate from the Formal Hypothesis series.