The Global Consciousness Project

Chris is simply wrong if he thinks XORing is some magic method of completely removing bias in noise-based RNG's, as Bancel's paper shows.

Just to be clear - perhaps you mean something different by bias, but what I mean is that the probability of the device producing a 0 is different from the probability of it producing a 1. With that definition, bias is necessarily removed by applying an XOR mask containing equal numbers of 0s and 1s, because exactly half the bits are thereby changed from 0 to 1 or vice versa. That means the probabilities have to become equal.

If you don't believe me, you can see that Bancel says the same thing:
"Theoretically, the RNGs output random 0 or 1 bits with equal probabilities. In real devices,
biases may occur due to component aging and other factors. Inherent device biases can
give rise to spurious correlations among the RNGs and they should be eliminated if
measured correlations are to be attributed to a GC effect. Accordingly, the devices employ
an XOR operation on the bits, which is a standard procedure for removing biases in RNG
bit streams. To simplify somewhat, the XOR essentially inverts, or "flips" half of the bits by
comparing them against a balanced XOR bit sequence. The procedure averages out any
persistent bias as long as half of the bits are inverted."
https://www.researchgate.net/publication...xploration

Yes, Bancel has found unexplained correlations through his statistical analysis, I have no reason to dispute that. But these are noise-based RNG devices, which are being used as  environmental measuring devices. And because they are noise-based, they pick up any environmental crap they are coupled to, and you can't tell whether the crap is having an effect on the devices bias towards 1's or towards 0's, or not, until you analyse the data. And even then, if you can't find anything to show such bias, you can't actually say bias isn't there.

When you actually read that 40 odd page document I linked to from SF's, with an open mind, and better understand the differences between different RNG's, their strengths and their weaknesses then, you might better understand how such tiny effects can creep into the data in a way that is not easily found.

With noise-based RNG devices you can't say where the noise is coming from, it might simply be that Bancel's statistical analysis has uncovered the predominant behavior of people during a 24 hour period, which is coupled to their use of electrical devices, which is coupled to electrical demand/usage, which is therefore coupled to the mechanism of a noise-based RNG, say... through it's power supply, and is affecting the bias of the device.

These devices don't need to be communicating with each other, they merely need to be affected by some change in the environmental noise that they use, and to which they are coupled, and which tends to be similar across the world, like people sleeping when it's dark, and working when it's light, when they eat, when they relax, when they are at home, when they are at work, when they heat the house, when they don't heat the house, when they watch game of thrones, when they stop watching game of thrones and boil the kettle for a cup of tea... etc. etc. etc....

 If the overall dataset is at expectation then this would indicate that such noise is not affecting the random input. If noise was continually resulting in non random data being produced over a large sample (and the GCP sample is huge) the results should vary wildly from expectation. 

I'm not sure you can have one without the other.

XORing doesn't do that in practice, can't do it, not on a noise-based RNG source... 

No matter what the source. It could be the Fibonacci sequence in binary, or a drawing of an elephant converted to digital form. If you apply a balanced XOR mask and average over all the possible positions of the mask relative to the bitstream, you will end up with equal numbers of 0s and 1s. It's just a matter of arithmetic.

I'm sorry Chris, we have a fundamental disagreement as to whether noise-based RNG devices are true random number generators... 

I'm not making a general statement about the characteristics of the devices. I'm making a specific statement that XORing with a balanced mask corrects exactly for an imbalance between the numbers of 0s and 1s in the bitstream.

For example, consider a series of 10 bits, like this: 
1110101101 (containing 7 1s and 3 0s)
There are more 1s than 0s there, and it was generated by a human rather than a random number generator, so there are probably too many 10s and 01s in there.

And consider a simple alternating mask like this:
0101010101...

There are two possible positions for the mask relative to that series of 10 bits:
0101010101 and 1010101010

When they're applied to the series we get:
1011111000 (containing 4 0s and 6 1s) and 0100000111 (containing 6 0s and 4 1s)

Averaging over the two positions of the mask, the expected number of 0s and 1s is 5 of each. The bias of the input has been exactly cancelled out. And that is bound to happen provided there are equal numbers of 0s and 1s in the mask.

I am NOT saying that if there are correlations between successive bits of the input that those will be eliminated by the XORing. Just that the bias will be eliminated, so that when a string of the bits are added up (as they were in the GCP) the mean of their sum will have the correct value, despite the bias.

[I only hope I got the arithmetic right there!]

OK, so you seem to accept that correlations amongst bits can't necessarily be removed by XORing the output from these noise-based devices... that's been confusing me.

But it's going to take a great deal of effort for me to look at the processes involved in how the GCP gathers it's data, in sufficient detail to actually understand their process, and go further than my claim that these noise-based devices have a bias problem, particularly when used as a measuring device...

However somethings is clearly not right in how I'm understanding things... Your Bancel quote says things, that seem - to me - to be at odds with what Koc & Stipcevic's RNG paper says (and you say?)  with regards to the limitations of XORing the biased output from noise based RNG's...

Koc & Stipcevic say...

"...if the raw bits exhibit strong correlations, simple procedures [XORing or Von Neuman] may not be sufficient to eliminate correlations among bits which can even be enhanced by simple de-biasing procedures or changed from short range to long range ones."

Your Bancel quote says...

"Inherent device biases can give rise to spurious correlations among the RNGs and they should be eliminated if measured correlations are to be attributed to a GC effect. Accordingly, the devices employ an XOR operation on the bits, which is a standard procedure for removing biases in RNG bit streams."


So I'm confused, one author seems to suggest that bias within the bits from noise-based RNG's can be eliminated by XORing, the other says XORing is not sufficient to eliminate bias. What am I misunderstanding here...?

I need to at least get this straight before I can go further...

Yes, that is a bit confusing, because while "bias" is used in the same sense in both those quotations (unequal probabilities of 1s and 0s), "correlations" is referring to two different things. In the first, it refers to correlations between successive bits produced by the same device, but in the second it refers to correlations between the sums of 200 bits produced by different devices at the same time (that is, the essence of what was observed in the GCP).

So the first is saying that removing the bias, using simple methods such as XORing, may not eliminate correlations between successive bits (and may even enhance it). 

But the second is concerned that if there is a similar bias in two different devices, then it may shift the mean of that sum of 200 bits away from the ideal value of 100 for both of them, and that would mimic a correlation between the two devices. Therefore the bias has to be removed by XORing, to prevent that happening.

Does that make sense?

I get what your saying.

Although I don't yet understand the relevance to the overall GCP process, of using the mean of the sum of every 200 XORed bits (you say 100 x 0's and 100 x 1's), so that the mean must by it's very nature always add up to 100, (or the specific details of how that occurs, which might be important).

Is there a detailed explanation of the GCP theory, and the exact process they use anywhere?

There's an outline of the way the data from the RNGs are processed on the GCP website here:
http://noosphere.princeton.edu/gcpdata.html
and also an overview with more on the hypotheses here:
http://noosphere.princeton.edu/science2.html

For more detail on Peter Bancel's analysis probably the best thing is his most recent paper:
https://www.researchgate.net/publication...xploration
which has additional technical material here:
https://www.researchgate.net/publication...al_details