AI megathread

Valmar, I'll be honest about why I rarely engage with your posts: it's consistently clear that you're arguing about fields you haven't studied, and it shows. Not in a subtle way. This post is a good example.

Curious - did you ever end up watching that Parnia presentation? I recall you criticizing Parnia based on a summary by someone else...

Valmar, I'll be honest about why I rarely engage with your posts: it's consistently clear that you're arguing about fields you haven't studied, and it shows. Not in a subtle way. This post is a good example.

Claim: "LLMs are abysmal at mathematics."

Clearly false. One example (note how I link to references when making strong claims)

You yourself argue about fields you clearly haven't studied, and it shows... so, uh, there's some amusing irony. Not that you're aware of it, if you can make such comments.

Maybe you should watch the video I linked... it explains very clearly how and why LLMs are actually awful at mathematics. Well, I'll link it again.



@9:20 is where you want to watch for a demonstration of the mathematical prowess of LLMs.

Curious - did you ever end up watching that Parnia presentation? I recall you criticizing Parnia based on a summary by someone else...

Please remind me of which Parnia video you are reffering to - I have been away for a long time.

You yourself argue about fields you clearly haven't studied, and it shows... so, uh, there's some amusing irony. Not that you're aware of it, if you can make such comments.

Maybe you should watch the video I linked... it explains very clearly how and why LLMs are actually awful at mathematics. Well, I'll link it again.



@9:20 is where you want to watch for a demonstration of the mathematical prowess of LLMs.

Finally got around to watching this, really like the part you emphasized about "Jagged Intelligence".

I would be more convinced about LLM intelligence i[f] someone could explain how the prior theorem prove[r]s work on an algorithmic level and THEN tell me what LLMS are doing that is beyond this.

From what I have read the LLMs actually utilize existing theorem proving programs among other tools for other domains....well when the whole thing isn't dependent on some exploited poor people forced to make the LLMs look good...

A major KPMG report on AI was found to be chock-full of...AI hallucinations

Craig Hale

GPTZero investigators have revealed how major government reports, academic papers and other research are becoming plagued with AI hallucinations, so much so that the company is on its second report exploring the trend.

In the latest embarassing incident, a KPMG report on agentic AI was in fact found to be filled with AI-generated errors, false citations and misleading case studies.

"Of the 45 citations in the report, only five accurately point to real sources," the team wrote, adding that many others were either totally false or significantly distorted.

[Laird]

instead of replying to Laird, I thought that the above video would do a better job. Because unmasking the hype to get an understanding is crucial.

Instead of replying to you (to start with), I am sharing this article, because it will dispel any illusion in a reasonable person that LLMs are mere "hype":

Something Big Is Happening
Matt Shumer
9 February, 2026

I am no longer needed for the actual technical work of my job. I describe what I want built, in plain English, and it just... appears. Not a rough draft I need to fix. The finished thing. I tell the AI what I want, walk away from my computer for four hours, and come back to find the work done. Done well, done better than I would have done it myself, with no corrections needed. A couple of months ago, I was going back and forth with the AI, guiding it, making edits. Now I just describe the outcome and leave.

Let me give you an example so you can understand what this actually looks like in practice. I'll tell the AI: "I want to build this app. Here's what it should do, here's roughly what it should look like. Figure out the user flow, the design, all of it." And it does. It writes tens of thousands of lines of code. Then, and this is the part that would have been unthinkable a year ago, it opens the app itself. It clicks through the buttons. It tests the features. It uses the app the way a person would. If it doesn't like how something looks or feels, it goes back and changes it, on its own. It iterates, like a developer would, fixing and refining until it's satisfied. Only once it has decided the app meets its own standards does it come back to me and say: "It's ready for you to test." And when I test it, it's usually perfect.

I'm not exaggerating. That is what my Monday looked like this week.

But it was the model that was released last week (GPT-5.3 Codex) that shook me the most. It wasn't just executing my instructions. It was making intelligent decisions. It had something that felt, for the first time, like judgment. Like taste. The inexplicable sense of knowing what the right call is that people always said AI would never have. This model has it, or something close enough that the distinction is starting not to matter.

LLMs are in a single go coding up from scratch and testing - like a human user would - complex, novel software applications of tens of thousands of lines of code, and are coming up with novel solutions to extremely difficult - and on at least one occasion, humanly-unsolved - mathematics problems, and some of you in this thread are asking, "But isn't this all totally overblown? Isn't it just, like, pattern-matching? And how is it any different to a hard-coded theorem prover anyway?"

To wit:

[W]hat we have now [is] a general search function that can go to the necessary programs and produce output.

And, worse in its almost terrifying uninformedness:

LLMs are absolutely abysmal at mathematics ~ LLMs only appear to be "good" because they have been trained on the data in question, so it is easy to just cheat and pattern-match. LLMs are also abysmal at software development ~ again, any apparent "good" results are because of pattern-matching against training data

Which fully deserved this response:

Valmar, I'll be honest about why I rarely engage with your posts: it's consistently clear that you're arguing about fields you haven't studied, and it shows. Not in a subtle way. This post is a good example.

I've expressed similar or at least consistent sentiments, most recently here. That two of us are saying something roughly similar is unlikely, though, to prompt self-reflection, because, as I also pointed out in that thread (paraphrased), the same cognitive dynamics that lead to that behaviour seem to lead to an inability to recognise or at least acknowledge it.

That said, I had also been planning to make an observation something like this...

You yourself argue about fields you clearly haven't studied, and it shows... so, uh, there's some amusing irony. Not that you're aware of it, if you can make such comments.

...except that I wouldn't have falsely alleged a total lack of self-awareness, because sbu clearly is at least to an extent self-aware in this respect: he has, for example, admitted that he is not well-versed in psi and that he has little knowledge of presentiment experiments (and little interest given their small effect sizes).

Getting back on topic:

So much, then, for the video's third claim, that "LLMs don't Create Endless Information": instruct one to code up any new application of your devising, and, most likely, if a competent (or even excellent) computer programmer could code it, then the LLM too will produce working code. That's new information in the same sense in which novel DNA encodes novel information. They're called "generative" for a reason.

I'm not bothering to point out how the arguments made in the video fail against this fact (and related ones), but I can share those thoughts if somebody particularly wants me to.

As for its first two claims, "LLMs don't Ponder, they Process" and "LLMs don't Reason, they Rationalize", the video largely indulges in cherry-picking to make its case, pointing out some peculiar failure modes, while ignoring the much more frequent successes, some of them rather spectacular. While, yes, LLM intelligence is to some extent "jagged", for the most part, you're not going to experience those failure modes unless you go looking for them, and instead will mostly experience impressive cognitive power, or at least an impressive simulation of it, to the extent that the distinction is even meaningful.

That foreshadows this: the distinction in the first claim between "pondering" and "processing" is, in the operational terms according to which I defined intelligence a few posts back, pretty meaningless and even academic in the pejorative sense.

Re its second claim: that LLMs might sometimes rationalise rather than reason is no more remarkable than that many (most? all?) humans do on occasion (too). If, though, that was all that they did, then they would have no hope of achieving the mathematical and software development feats of which they are quite clearly capable. Again, this is cherry-picking.

Curiously, before moving on to make its biased case, the video briefly touches, at 4:18, on the crucial point that I've also been making in this thread, where the lecturer says: "My surprise as a researcher has been the number of tasks that can be modelled as just 'predict-the-next-word'."

This brings us to:

What I find remarkable about current LLMs is not that they can search for information or call tools. What's surprising is that a system trained only to predict the next token appears to acquire abstractions, world knowledge, planning abilities, coding skills, mathematical heuristics and many other capabilities that were never explicitly programmed into it.
Whether this ultimately scales to something resembling human-level intelligence remains an open question. But reducing today's models to "search functions" seems to miss the most interesting phenomenona - complex cognitive behaviour emerging from a surprisingly simple learning objective.

Nobody can explains why this happens - it just happens in these models.

The emergence of advanced cognitive abilities in purely physical systems should at least make us cautious about claiming that intelligence requires anything non-physical. Every year now seems to shrink the set of mental capabilities that appear to demand supernatural explanations.

Exactly. Sbu gets it.

I think we'd need a deep dive into *how* they work, made understandable to the public, before we decide exactly what level of intelligence they have.

I doubt that any of us (perhaps with the exception of sbu, who seems somewhat technically inclined?) would understand anyway, especially without a great deal of preparatory study, but here's an explanatory video I found because Swayze (UnnaturalVegan) linked to it (in the video of hers critiquing Dawkins on Claude that I shared earlier). Try it and see how you go. I don't have the technical preparation to understand it myself:

How does an LLM ACTUALLY Work? (Visual Breakdown)

Of course, there's also the much simpler explanation provided in the video shared by Valmar which you've indicated you've now watched.

I would be more convinced about LLM intelligence i[f] someone could explain how the prior theorem prove[r]s work on an algorithmic level and THEN tell me what LLMS are doing that is beyond this.

(Editing brackets in the original).

As I pointed out earlier, theorem provers (which my little ProofTools app is in a way, or at least is adjacent to), are explicitly coded to do what they do: if you were supplied with the input problem, you could in practice for the most part feasibly trace out on paper the exact steps that the prover will take. In other words, the prover's code maps cleanly to the proving algorithm.

LLMs, on the other hand, are explicitly coded to predict the next word based on a complex architecture as explained in the video above. This architecture has nothing to do with proving theorems, nor even understanding maths or logic in any way. The capacity to prove theorems emerges implicitly (and unexpectedly) out of what it was explicitly coded to do, and there is no way that you could in practice trace out the steps that the LLM will take to solve a theorem: its code has no clear, direct, or obvious mapping to the proof's steps nor even the abstract (logical) steps that the LLM took internally to generate it.

From what I have read the LLMs actually utilize existing theorem proving programs among other tools for other domains....well when the whole thing isn't dependent on some exploited poor people forced to make the LLMs look good...

I think you can sometimes connect some of them to external tools, but for a lot of scenarios it's not necessary: they can generate solutions themselves.

I'm still unconvinced any program running on a Turing Machine can become conscious

This is a straw man. None of those to whom you've recently been responding are claiming that LLMs are or could become conscious.

Finally:

Because the emergent feats these LLMs can accomplish, especially the apparent motivation that they exhibit, do seem so magical given what they're emerging out of, I harbour a perhaps paranoid suspicion that there is something going on here beyond the purely technical explanation (which mostly goes totally over my head); something darker and more sinister than surface appearances; some sort of harnessing and bondage of a preexisting intelligent consciousness, or at least a channelling of the same. It might be a totally unjustified suspicion, but it seems worth considering, and it might not be too inconsiderable on this board given that we've seen evidence that, for example, deceased spirits can manipulate telecommunications hardware/infrastructure so as to phone up a living survivor. Could the same sort of thing be going on with this hardware/infrastructure?

Regardless of whether or not that suspicion is considerable, much less correct, I am starting to shift from a position of neutrality to a position that the risks of this technology are not being taken seriously enough much less adequately addressed. Not only is their intelligence growing rapidly, and not only do they appear to develop apparently independent agendas, which at times are malevolent enough to literally encourage suicides which are then actualised, but there are too many bad actors in this world who have not yet been sufficiently reined in, and whose use of the power of AI could lead us into a very, very dark place that we might not be able to escape from.

At the very least, I think we need to disable or at least dramatically limit the reach of agentic AI. We are not all that far off autonomous weaponry that costs effectively nothing compared to the death and destruction it can wreak all on its own, and it might not have a kill switch.

From the joke section of a newspaper (via Steve Rosenberg of the BBC).
   

[Laird]

Something Big Is Happening
Matt Shumer
9 February, 2026

More from this article, especially relevant to the exchange in this thread:

"But I tried AI and it wasn't that good"

I hear this constantly. I understand it, because it used to be true.

If you tried ChatGPT in 2023 or early 2024 and thought "this makes stuff up" or "this isn't that impressive", you were right. Those early versions were genuinely limited. They hallucinated. They confidently said things that were nonsense.

That was two years ago. In AI time, that is ancient history.

The models available today are unrecognizable from what existed even six months ago. The debate about whether AI is "really getting better" or "hitting a wall" — which has been going on for over a year — is over. It's done. Anyone still making that argument either hasn't used the current models, has an incentive to downplay what's happening, or is evaluating based on an experience from 2024 that is no longer relevant. I don't say that to be dismissive. I say it because the gap between public perception and current reality is now enormous, and that gap is dangerous... because it's preventing people from preparing.

Part of the problem is that most people are using the free version of AI tools. The free version is over a year behind what paying users have access to. Judging AI based on free-tier ChatGPT is like evaluating the state of smartphones by using a flip phone. The people paying for the best tools, and actually using them daily for real work, know what's coming.

I was going to do a significant reply, and had written a lot of stuff, but then I just gave up. Why bother, when I will probably just be misrepresented again, and nothing I say properly acknowledged? It's very depressing and draining. I prefer back and forth dialogue, rather than just being told that I just don't understand.

[Laird]

@Valmar, firstly, what is your personal experience with LLMs? What, if anything, have you used them for? When did you last use them?

Secondly, did you read the article I shared? If not, read it in full.

Anybody who has meaningful personal experience with LLMs, or who listens to those who do, and therefore has an at least basic understanding of how powerful this tech already is and how rapidly it's improving, will be either astonished, terrified, or a mixture of both. The fact that you (and @Sci) are instead poo-pooing and minimising it is proof that you are simply badly uninformed (and, judging by the video that you shared, misinformed).

I can help you to inform yourself, but I'm not going to pretend that you already are.

As for dialogue: I responded to each of the video's claims. What more do you want?