AI megathread

Those tests using fairly straightforward maths problems are revealing, because the mistakes and fabrications can be identified and shown for what they are. But it's worrying if the AI tool is being used to produce outputs relating to more loosely-defined real-world subjects. It may be much less obvious when the output is simply nonsense, because it may appear plausible.

OpenAI’s dirty December o3 demo doesn’t readily replicate

Gary Marcus

Later, after I wrote that piece, I discovered that one of their demos, on FrontierMath, was fishy in a different way: OpenAI had privileged access to data their competitors didn’t have, but didn’t acknowledge this. They also (if I recall) failed to disclose their financial contributions in developing the test. And then a couple weeks ago we all saw that current models struggled mightly on the USA Math Olympiad problems that were fresh out of the oven, hence hard to prepare for in advance.

Today I learned that the story is actually even worse than all that: the crown jewel that they reported on the demo — the 75% on Francois Chollet’s ARC test (once called ARC-AGI) doesn’t readily replicate. Mike Knoop from the ARC team reports “We could not get complete data for o3 (high) test due to repeat timeouts. Fewer than half of tasks returned any result exhausting >$50k test budget. We really tried!” The model that is released as “o3 (high)” presumed to be their best model, can’t readily yield whatever was reported in December under the name o3.

The best stable result that ARC team could get from experimenting with the latest batch of publicly-testable OpenAI models was 56% with a different model called o3-medium, still impressive, still useful, but a long way from the surprising 75% that was advertised.

And the lower 56% is not much different from what Jacob Andreas’s lab at MIT got in November. It’s arguably worse; if I followed correctly, and if the measures are the same, Andreas lab’s best score was actually higher, at 61%.

Four months later, OpenAI, with its ever more confusing nomenclature, has released a bunch of models with o3 in the title, but none of them can reliably do what was in the widely viewed and widely discussed December livestream. That’s bad.

Forgive if me I am getting Theranos vibes.

[Laird]

A couple of recent videos on Michael Berman's YouTube channel have documented two interesting, related new developments (neither of which I've - yet? - read in the originals in full), one a paper and the other an announcement.

They each, to different degrees, concern the potential for the independent, self-directed learning and even self-evolution of AI systems.

Michael's video re the paper Absolute Zero: Reinforced Self-play Reasoning with Zero Data is:

New "Absolute Zero" Model Learns with NO DATA.

His video re the announcement AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms is:

Self-Improving AI is here... (Alpha Evolve).

[Laird]

Following on from that, but in a different vein, our ABC has published a couple of interesting news articles related to AI in the past few days:

Visa and Mastercard are developing artificial intelligence 'agents' to spend your money for you by Luke Cooper on the 18th of May, 2025.

Ivana Tranchini, Visa's head of client engagement in Australia, told the ABC that the Intelligent Commerce platform will allow for consumers to manually action their own security checks so they have the final say on when their money is spent.

"It comes down to what the consumer wants. I could decide I want to get an 'OK' opportunity just before purchase is made based on all of the different parameters I've already put in place, so I can control that," she said.

"Or, I could say, 'Actually, if it's below $5,000 I'm happy for you to purchase it.'"

I'm not so sure how soon this is going to catch on, if ever. On the other hand, this part of it might get more traction (editing brackets in the original):

Mastercard told the ABC it had a five-year plan to "transform payments in Australia" that includes a push to get rid of traditional 16-digit bank card numbers and replace them with these dynamic one-time tokens.

Mr Long said that part of Mastercard's recent announcements sounded positive.

"You'll be able to go, 'I want a card number that is only used for streaming services' [and] boom, you'll get a special card number. You don't have to apply for a new card or a new account, you just have another card number on your existing account," he said.

Then there's Young Australians using AI bots for therapy by April McLennan, from yesterday, the 19th of May, 2025.

"A lot of people that I know around my age group are like 'yeah that's actually really clever', and then you don't get that guilt of venting to people that you know.

"I wouldn't be surprised with the rise of this use, if people started bringing out almost targeted AIs that are for this."

For Gracie, she doesn't know if she will ever be able to stop using AI for therapy.

"I just feel like, at this point now, because I'm a very empathetic person, and I've actually formed an emotional bond with it.

"I don't want to leave it behind now, like I just want to keep bringing it forward."

Has anyone tried to test an AI by engaging in a discussion about a paranormal phenomena and seeing if the relies on pseudoskeptical dogma and if so, what happens when you try to argue with actual evidence?

[Laird]

Has anyone tried to test an AI by engaging in a discussion about a paranormal phenomena and seeing if the relies on pseudoskeptical dogma and if so, what happens when you try to argue with actual evidence?

Based on what I remember (only a little) from reading the transcript a while back, Alex Tsakiris tried out something like that in this episode:

AI Walks It Back |605|

I can't find the transcript again though, so you might have to listen to it.

Another, later, episode in which he seems to try something similar and even more relevant, but which I haven't read or listened to (yet?), so can't confirm, is:

AI Exposes Truth About NDEs |632|

Has anyone tried to test an AI by engaging in a discussion about a paranormal phenomena and seeing if the relies on pseudoskeptical dogma and if so, what happens when you try to argue with actual evidence?

Try this yourself. Google now responds to several sentences of text with an AI response. Grok on X.com is free, and just limits the number of interactions (questions) in every 24h period

For instance carefully evolving the text of a long question to this in Google:

"Why is there a temporal correlation between the recalled 'apparently' veridical Experience during a medical-type NDE OBE, and the period around the time of their cardiac arrest and resuscitation. Many of these medical-type NDEs with 'veridical' OBE recall occur on the operating table whilst the patient is under anaesthesia? The temporal relationship between the patients recalled visual information and the period of cardiac arrest, when also combined with the patient sometimes being under anaesthesia is a real puzzle."

Got this response:

The correlation between recalled NDE experiences and the period of cardiac arrest, even when a patient is under anesthesia, is a complex issue that remains a subject of ongoing research and debate. While some explanations focus on the potential for brief periods of brain activity or hormonal releases during cardiac arrest, others explore the possibility of pre-existing beliefs and cultural influences shaping these experiences, according to the University of Virginia School of Medicine.

Here's a breakdown of the key aspects and potential explanations:

1. The Puzzle:

Temporal Correlation:
Many patients who experience near-death experiences (NDEs), especially those with "veridical" OBEs (out-of-body experiences) during medical procedures like cardiac arrest, report details of their resuscitation, which they could not have otherwise observed.

Under Anaesthesia:
This is particularly perplexing when patients are under anesthesia, as anesthesia is designed to suppress brain activity.

Brain Activity:
Conventional understanding suggests that the brain ceases to function shortly after cardiac arrest due to lack of oxygen, making it difficult to reconcile the reported cognitive experiences with this physiological state.

2. Potential Explanations:

Brief Periods of Brain Activity:
Some studies have found that during CPR, patients exhibit spikes in brain activity, potentially linked to thought and memory. These spikes could be brief but intense, occurring during the period of cardiac arrest or shortly after resuscitation.

Hormonal Releases:
Research suggests that during stressful events like cardiac arrest, the body may release neuroprotective agents, similar to ketamine, which could influence consciousness and perception.

Pre-existing Beliefs and Cultural Influences:
Some theories suggest that NDEs are shaped by an individual's prior beliefs and cultural expectations about death and the afterlife.

Psychological Factors:
The threat of death and the intensity of the experience could lead to vivid and emotionally charged memories, potentially influencing the details of the recalled experience.

Veridical OBEs:
Some NDEs involve veridical OBEs, where individuals report accurately recalling events happening around them, even at a distance, which is difficult to explain with purely physicalist theories of consciousness.

3. Ongoing Research and Challenges:

    Need for Prospective Studies:
    Research on NDEs is ongoing, and more prospective studies are needed to gain a better understanding of the underlying mechanisms.

Methodological Challenges:
Studying NDEs presents methodological challenges, as it's difficult to control for various factors like patient demographics, medical history, and individual beliefs.

Ethical Considerations:
The nature of NDEs raises ethical considerations, particularly regarding the potential for patients to experience distress or psychological trauma related to their experiences.

In conclusion, the temporal correlation between recalled NDE experiences and the period of cardiac arrest is a complex phenomenon that requires further investigation. While various theories attempt to explain these experiences, the exact mechanisms and underlying processes remain a subject of ongoing research and debate.




Posing the same question to public Grok on X.com

The temporal correlation between near-death experiences (NDEs) with out-of-body experiences (OBEs) and cardiac arrest likely stems from brain activity surges during CPR, including gamma and theta waves, which may create vivid perceptions. Oxygen deprivation and neurotransmitter release can also induce dissociative states, explaining the timing. About 10-20% of cardiac arrest survivors report such experiences, some with verified details, suggesting possible consciousness during clinical unconsciousness. However, these could be brain-driven, as similar experiences occur under anesthesia or drugs. The debate persists, with some proposing non-physical consciousness, but evidence leans toward neurological causes. More research is needed to clarify.





Posing the same question to private Grok on X.com

The temporal correlation between recalled veridical experiences during near-death experiences (NDEs) with out-of-body experiences (OBEs) and the period of cardiac arrest or resuscitation, particularly under anesthesia, is indeed a perplexing phenomenon. Below, I’ll break down the key aspects of this puzzle and explore possible explanations, drawing on available research and reasoning, while acknowledging the speculative nature of some interpretations due to the complexity and limited empirical data.

Key Observations

Temporal Correlation: Many NDE reports describe vivid, detailed perceptions (e.g., visual or auditory experiences) that seem to align with events occurring during cardiac arrest or resuscitation. These experiences often include details that the patient could not have known through normal sensory means, termed "veridical" because they are later corroborated by others (e.g., medical staff or family).

Anesthesia Context: Some NDEs occur under general anesthesia, a state where consciousness and sensory processing are suppressed, making it surprising that patients report vivid experiences during this period.

Cardiac Arrest Context: During cardiac arrest, cerebral blood flow and oxygen delivery are severely compromised, leading to a state where normal brain function, including consciousness, is presumed to be absent or minimal. Yet, some patients report structured, lucid experiences during this time.

Possible Explanations

The temporal correlation and the occurrence of veridical OBEs under anesthesia and cardiac arrest have sparked various hypotheses, ranging from neuroscientific to metaphysical. Here are the main perspectives:

1. Neuroscientific and Physiological Hypotheses
Residual Brain Activity: Even during cardiac arrest, some residual brain activity may persist for a short period (seconds to minutes) due to incomplete cessation of neural function. Studies, like those by Sam Parnia (e.g., the AWARE study), suggest that brain activity might not completely stop immediately after cardiac arrest. This could allow fragmented sensory processing or memory formation, though it’s unclear how this produces vivid, structured experiences.

Anesthetic Awareness: In rare cases, patients under anesthesia may experience "anesthetic awareness," where they are partially conscious despite appearing sedated. However, this doesn’t fully explain veridical OBEs, as patients often report details (e.g., seeing events from above the operating table) that exceed normal sensory access.

Memory Reconstruction: The brain may reconstruct experiences post-resuscitation, blending fragmented sensory input, imagination, or prior knowledge. The temporal correlation could arise from the brain "timestamping" these reconstructions to the period of crisis, even if the experiences are formed later. This is supported by research on confabulation and false memories in neurological conditions.

Hypoxic or Hypercapnic Effects: Cerebral hypoxia (low oxygen) or hypercapnia (high CO2) during cardiac arrest can induce altered states of consciousness, including vivid hallucinations or dream-like states. These might be interpreted as OBEs, though their veridical nature (e.g., accurately describing operating room events) remains harder to explain.

Drug Effects: Anesthetic drugs like ketamine or propofol can induce dissociative or hallucinatory states. While these drugs typically suppress consciousness, their effects during recovery or in atypical cases might contribute to NDE-like experiences.

2. Psychological and Cognitive Hypotheses
Expectation and Cultural Influence: NDE content often aligns with cultural or personal expectations (e.g., seeing a tunnel or deceased relatives). The temporal correlation might reflect the brain’s attempt to make sense of a life-threatening event by constructing a narrative tied to the moment of crisis.

Time Perception Distortion: During extreme stress or altered states, the perception of time can be distorted. Patients might perceive events as occurring during cardiac arrest, even if the memories are formed during recovery or partial consciousness.

Veridicality as Coincidence or Bias: Some veridical reports might result from coincidence, vague descriptions being interpreted as accurate, or confirmation bias by researchers or witnesses. For example, a patient’s description of a “doctor in blue scrubs” might seem veridical but is a common detail in operating rooms.

3. Non-Materialist Hypotheses
Consciousness Beyond the Brain: Some researchers and philosophers (e.g., Pim van Lommel) propose that consciousness may not be entirely dependent on brain function, suggesting that NDEs reflect a non-local or transcendent aspect of consciousness. Veridical OBEs, especially under anesthesia, are cited as evidence, as patients report perceiving events when brain activity is presumed minimal or absent. This view posits that the temporal correlation reflects consciousness operating independently during the cardiac arrest period.

Shared Consciousness or Psi Phenomena: Some speculate that veridical perceptions could involve accessing information through non-sensory means (e.g., telepathy or clairvoyance), though this lacks empirical support and is controversial within mainstream science.

Spiritual or Metaphysical Explanations: Many NDE experiencers interpret their experiences as evidence of a soul or afterlife, with the OBE reflecting a temporary separation of consciousness from the body. The temporal correlation would then align with the moment of closest proximity to death.

4. Challenges with Anesthesia and Veridicality
Anesthesia’s Effect on Consciousness: General anesthesia typically induces a state of unconsciousness by disrupting neural networks (e.g., via GABAergic drugs like propofol). The fact that some patients report detailed experiences during this period challenges the assumption that consciousness is fully suppressed. Research into anesthetic mechanisms is ongoing, but it’s possible that certain brain regions remain active enough to process or store information in rare cases.

Veridical Details Under Anesthesia: Reports of patients accurately describing operating room events (e.g., specific conversations or tools used) while under anesthesia are particularly puzzling. For example, cases like Pam Reynolds’ NDE, where she described surgical details despite being under deep anesthesia and induced hypothermia, remain difficult to explain through conventional neuroscience.

Research and Evidence
AWARE Studies (Parnia et al.): The AWARE I and II studies investigated NDEs during cardiac arrest, finding that some patients reported awareness and memories that seemed temporally linked to the resuscitation period. However, no definitive veridical OBE was confirmed (e.g., patients didn’t report seeing hidden targets placed in operating rooms).

Van Lommel’s Study (2001): A prospective study in The Lancet found that 18% of cardiac arrest survivors reported NDEs, with some describing veridical details. The study suggested that these experiences challenge purely physiological explanations, as they occurred during presumed unconsciousness.

Greyson Scale and NDE Features: The Greyson NDE Scale highlights common features like OBEs, vivid perceptions, and a sense of timelessness, which align with the temporal correlation but don’t explain its mechanism.

Neuroscience Limitations: Current brain monitoring (e.g., EEG) during cardiac arrest or anesthesia may not capture all forms of neural activity, especially in deeper brain structures like the brainstem or thalamus, which could play a role in consciousness.

Why It’s a Puzzle
The combination of temporal correlation, anesthesia, and veridicality is challenging because:
Neuroscience Gap: We lack a complete understanding of consciousness, especially under extreme conditions like cardiac arrest or anesthesia. Standard models assume consciousness requires robust brain activity, yet NDEs suggest otherwise.

Veridicality: Accurate details reported by patients (e.g., specific conversations or visual scenes) are hard to reconcile with sensory deprivation or brain dysfunction.

Timing: The precise alignment of recalled experiences with the cardiac arrest period suggests either real-time perception or a remarkable ability of the brain to retroactively construct accurate narratives.

Anesthesia Paradox: Anesthesia suppresses sensory input and memory formation, yet some patients report structured, detailed experiences, raising questions about how and when these memories are formed.

Current Status and Future Directions
Skeptical View: Many neuroscientists argue that NDEs are likely a product of brain processes under stress, with veridicality explained by coincidence, memory reconstruction, or subtle sensory cues (e.g., hearing conversations during light anesthesia).

Open Questions: Proponents of non-materialist views argue that NDEs challenge reductionist models of consciousness, but these claims require extraordinary evidence, which is currently lacking.

Research Needs: More rigorous studies, like AWARE II, are needed to test veridicality (e.g., using hidden targets) and measure brain activity during cardiac arrest with advanced tools (e.g., fMRI or invasive EEG, if feasible).

Conclusion
The temporal correlation between veridical OBEs and cardiac arrest, especially under anesthesia, remains a profound mystery. Neuroscientific explanations point to residual brain activity, memory reconstruction, or drug effects, but these struggle to fully account for veridical details and the vividness of experiences. Non-materialist hypotheses, while intriguing, lack empirical grounding and are not widely accepted. The puzzle persists due to gaps in our understanding of consciousness and the limitations of current research methods. Future studies with better monitoring and controlled conditions may shed light, but for now, the phenomenon challenges both scientific and philosophical frameworks.



-----------------------------


Public vs Private Grok is profoundly different. Public Grok barely gives any latitude to ideas without hard scientific evidence.

[Laird]

Try this yourself.

For Jim's benefit (and others), here are a few of the major chatbots that can be used for free and (mostly) without logging in (although you can also create an account if you want to keep a history of your conversations):

ChatGPT (by OpenAI, funded by Microsoft):
https://chatgpt.com/

Grok (Elon Musk's):
https://x.ai/grok

Gemini (by Google):
https://gemini.google.com/app

DeepSeek (by the eponymous Chinese company):
https://chat.deepseek.com/

Claude (by Anthropic)*:
https://claude.ai/

* I haven't actually used Claude before, and it doesn't seem to be accessible without first creating a (free) account. The others I've used and they all work fine for free without logging in.

[Edited to add: feel free to respond with any others, and I'll add them to this post]

https://instapundit.substack.com/p/scary...t-scary-ai

What’s more, an AI system will be able to draw on information regarding billions of people and personalities, and it will be able to use that information to be extremely persuasive to anyone it talks to. You may think you’re smart and skeptical, but your intellect is unlikely to be a match for a being that is 100 times smarter than you, and that knows the emotional and intellectual weaknesses of every human like you.
...
It’s a game where the only winning move is not to play. That is, don’t engage, don’t talk, or argue, or listen. That way lies tragedy.

"the only winning move is not to play" is a quote which at least for me is associated with the 1983 film Wargames.

An entertaining film where both a computer and the humans seem unable to distinguish between reality and a simulation. The computer understandably so - perhaps? Though it is transparently obvious which is which to at least some of the humans.