AI megathread

ChatGPT Brain Rot Is Real (And Getting Worse)

ChatGPT went down and people couldn't function. But the real problem goes much deeper than a service outage. From hikers getting lost after relying on ChatGPT to people uploading their entire closets for AI fashion advice, we are outsourcing every single thought to the LLM. Recent studies suggest we are losing our critical thinking and our ability to problem solve by over relying on ChatGPT.

Research:

Microsoft: https://www.microsoft.com/en-us/research...survey.pdf 

MIT: https://www.media.mit.edu/publications/y...n-chatgpt/ 

Wharton: https://papers.ssrn.com/sol3/papers.cfm?...id=5104064

Chapters:
00:00 Intro
00:43 Cannot Bake Without ChatGPT
01:50 ChatGPT Travel Information
02:30 Fact Checking NASA
03:33 Cheating in Interviews
04:32 Teachers: AI is Destroying Learning
05:40 AI and Calculator Comparison
05:56 ChatGPT and Universities
06:36 Cheating on Exam with AI
07:13 Relying on ChatGPT for Hiking
07:42 Leveling Up with ChatGPT
08:24 Wharton Research
08:50 MIT Research
09:25 Microsoft Research
09:41 AI Paradox
10:24 Everyone Needs a Side Hustle
11:41 What Happened to Honing Your Craft?
12:23 AI Folks Discover “Thinking”
12:37 ChatGPT for Wardrobe

Nobody Actually Wants AI Anymore

People often compare AI to the internet, but there's one big problem with that comparison: people naturally adopted the internet as it became available over the years. AI, on the other hand, is being forced into every crevice of our lives and the response is increasingly "we don't want AI."

Sources:

AI Assistance Reduces Persistence and Hurts Independent Performance: https://arxiv.org/pdf/2604.04721

Harvard Brain Fry: https://hbr.org/2026/03/when-using-ai-le...-brain-fry

Writer’s AI adoption in the enterprise: https://www.morningstar.com/news/busines...t-adopt-ai

Chapters:
00:00 Intro
00:26 Stop Telling Me to Use AI
01:21 AI Replacing Customer Service Reps
02:22 AI is in Everything
03:11 Pinterest is Ruined By ChatGPT AI Images
03:33 Brands Facing Backlash for AI Content
03:57 Prada AI Slop
04:36 AI is Ruining Jobs
05:16 Usage of AI Reduces Our Own Skills
05:47 Companies Enforcing and Tracking AI Usage
06:15 Uber’s $3.4 Billion AI Budget
06:46 AI Agents for AI Slop
07:41 Using AI for the Most Basic Tasks
08:12 Software Engineers and AI Fatigue
09:07 AI Agents Causing Brain Fry
09:28 Executives Not Seeing Return on AI Investment
10:01 Students Boo at Commencement Speaker
10:52 Former Google CEO Booed at Graduation
11:35 Volume Has Become the Illusion of Value
12:06 Outro

[Laird]

About that Matt Shumer post that has nearly 50 million views

Gary Marcus

You miss the point, Sci. It's not important whether or not Matt's post was overly optimistic. It's not even important that LLMs still have limitations, including reasoning failures, rationalisations, and hallucinations. The point is that they have reached a point where they are doing truly remarkable things, like writing, and testing like a human would, ten-thousand line apps in a single go, and solving meaningful, humanly-unsolved maths problems, and that they continue to improve.

As I pointed out in another thread, three years ago, ChatGPT couldn't solve a deceptively simple maths problem after multiple attempts, providing mistaken reasoning on each attempt; two days ago, the latest model solved it straight away, in a few seconds, providing the correct mathematical reasoning immediately. That's the free model too; the paid models are even more capable.

The idea that these achievements are based on mere pattern-matching, as though each prompt posed an interchangeable, cookie-cutter problem, is absurd. Likewise is the idea that they do all this by rationalising and not by reasoning. Most absurd is the idea that LLMs are "abysmal" at both software development and mathematics.

One thing that seems worth adding as a caveat though is that a lot of intelligence, including human intelligence, does seem to be related to pattern recognition. We can "recognise", for example, when one technique from one field could be used to solve a problem in another field (it "matches"), and we go ahead and apply it in that other field (as a "pattern"). LLMs do seem to do this too. This is on a whole other level though than "apply this near-verbatim recipe to this near-verbatim, stock problem", which, it seems, is what Valmar and perhaps even yourself seem to think is all LLMs are capable of.

Let me ask, then, as I asked of Valmar: to what extent have you yourself used LLMs, Sci?

You miss the point, Sci. It's not important whether or not Matt's post was overly optimistic. It's not even important that LLMs still have limitations, including reasoning failures, rationalisations, and hallucinations. The point is that they have reached a point where they are doing truly remarkable things, like writing, and testing like a human would, ten-thousand line apps in a single go, and solving meaningful, humanly-unsolved maths problems, and that they continue to improve.

Did anyone here say they programs aren't useful at all? We're integrating AI use at work, though prior flaws made us wary.

I'm just not sure what metric "truly remarkable" is based on.

I find it odd that you will post someone's article, and then when I post a critique the article is no longer important?

I guess what confuses me is what is it you want me to admit. It seems I have to believe something "truly remarkable" is happening, but even you don't believe the program is conscious...just intelligent for mysterious reasons that may or may not have to do with spirit channeling.

It just seems to me that skeptics like @sbu who have no problem believing a medium or NDEr will embellish or even lie for money and fame suddenly drop their skeptical sense when billions to trillions of dollars for AI companies is on the line. Now word prediction has miraculous properties, though of course instead of considering Platonism or Idealism he jumped to the idea the this - for unexplained reasons of course - supports the Physicalist faith.

For myself, I've posted for years at this point on the history of AI where people rushed to believe in thinking programs only to realize later they'd jumped the gun....where AI companies in modern times have been shown to be deceptive....where studies show slight shifts in inputs can confound LLMs...

As such, as a true skeptic rather than an evangelical for the Physicalist faith...I think skepticism is warranted that some program is exhibiting true intelligence...

@Valmar

An article on Jagged Intelligence by Melanie Mitchell that might be of interest.

THAT LLMS APPEAR to understand language, though, does not mean they actually understand it as humans do. Indeed, while AI boosters have touted the superhuman capabilities of LLMs and their astounding successes, other AI users have noticed, and reported on, their puzzling, unhumanlike failures, which have not gone away as these systems have progressed. How can a system that has exceeded human performance on advanced math problems sometimes fail at simple elementary-school-level problems? Why do these systems answer a question perfectly when it is worded one way but struggle when it is worded in a different but (to a human) equivalent way? How can a system that generates accurate and incisive summaries of books also produce similarly confident and authoritative-sounding summaries of nonexistent titles? How can a system that has been extensively trained to refuse dangerous requests be easily fooled by “prompt engineering” into cheerfully providing the prohibited information?

An article on Jagged Intelligence by Melanie Mitchell that might be of interest.

Indeed... it covers many similar points as the video I posted, so perhaps this could be one source he used. Or maybe they happened upon similar to same sources, as these papers present major issues for the industry:

Indeed, despite the ubiquitous use of such benchmarks for assessing AI capabilities, there is a sense in the AI research community that, in the words of one recent paper, “benchmarking is broken.” So why might an AI system’s performance on benchmarks oversell its real-world capabilities? One widespread problem is what’s called “data contamination”: an AI system might have had some of the benchmark items, or very similar kinds of questions, in its training data, making the test one of memorization rather than of reasoning. OpenAI’s GPT-4, for instance, performed exceptionally well on a computer programming benchmark when it was answering questions published before 2021, when the model’s training was stopped. On problems published after 2021, GPT-4’s performance declined sharply—a strong indication that the earlier problems were in the system’s training data.

So, not "jagged intelligence" so much as the LLM algorithm pattern-matching according to tokens existing the training data.

Memorization is not something we class "intelligence" when it comes to school children figuring out answers to exams, so why should that be granted to an LLM? It's not like we have an actual meaningful definition of "intelligence" when applied to conscious beings, only a vague generalization around mental powers of creativity and problem-solving by applying existing skills to new domains. But then we also have little idea of how to properly define "creativity". What does it actually mean to "solve" a problem? On the surface, it appears simple, but it is not so... the best philosophers to this day can't really agree on it on what these words or the concepts they point to mean. Even the Greeks couldn't figure it out. That shouldn't allow just anyone to confer "intelligence" onto something non-conscious just because it is difficult to define.

As documented by the cognitive linguist and philosopher George Lakoff and his fellow collaborators, humans tend to frame abstract concepts as metaphors. This is especially true when we struggle with how to conceptualize something new, be it the internet, social media, or AI. Our chosen metaphors can strongly affect how we interact with these technologies, as well as how we legislate and regulate them. Today, users often view chatbots, which interact with them using the first-person “I,” as companions, therapists, or even romantic partners, roles that cannot be played by a system understood to be more like a library or bureaucracy. The recent rise of AI “agents,” often positioned as “personal assistants,” has only solidified the metaphor of AI as a kind of individual mind.

I guess this quote puts it more neatly than I ~ we have a tendency to use metaphors, but then there is an inherent danger of confusing these metaphors with the literal idea. Such as with intelligence. We confuse appearances of "intelligence" with the real deal, simply because the same word is used for entirely distinct concepts and ideas. Metaphors are only necessary because we lack good, existing words, I think, along with proper conceptualizations.

But that also allows malicious marketeers to deliberately conflate different ideas to create false concepts and ideas by which they can then sell a product to the unwary.

[Laird]

@Sci, since you dodged my question, I assume that, like @Valmar, you have little to no personal experience with LLMs. That figures.

I've made the necessary points, and I won't belabour them. I'll simply add some observations:

It seems that just as @sbu's metaphysical beliefs lead him to reflexively dismiss the significance (and the very existence) of very real phenomena like UFOs and psi given that they challenge those beliefs, so your (and @Valmar's) metaphysical beliefs lead you (both) to reflexively dismiss the significance (and the very existence) of the very real intelligence of LLMs given that it challenges those beliefs.

I find this quite curious given that to a meaningful extent, @nbtruthman and I share your metaphysical beliefs, yet both of us, having explored LLMs a little, are quite willing to affirm their remarkable intelligence (while of course at the same time acknowledging that it has certain - sometimes very strange and face-palm - failure modes).

Another word on top of "curious" is "disappointing". I expect better from you.

It seems that just as sbu's metaphysical beliefs lead him to reflexively dismiss the significance (and the very existence) of very real phenomena like UFOs and psi given that they challenge those beliefs, so your (and Valmar's) metaphysical beliefs lead you (both) to reflexively dismiss the significance (and the very existence) of the very real intelligence of LLMs given that it challenges those beliefs.

I find this quite curious given that to a meaningful extent, nbtruthman and I share your metaphysical beliefs, yet both of us, having explored LLMs a little, are quite willing to affirm their remarkable intelligence (while of course at the same time acknowledging that it has certain - sometimes very strange and face-palm - failure modes).

Well... a challenge for you ~ what precisely does the word "intelligence" mean to you? With as much detail as possible. I am curious, because I'm not sure you have actually detailed what the word means when you use it.

[Laird]

Well... a challenge for you ~ what precisely does the word "intelligence" mean to you?

When out of arguments, resort to semantics.

I'm not sure you have actually detailed what the word means when you use it.

Then you haven't been paying attention.

I notice that you didn't deny my observation.

When out of arguments, resort to semantics.

Please stop accusing me of things. I just asked a question.

Then you haven't been paying attention.

Hmmmmm

Let's start with an operational definition like Alan Turing did and say it's the property of an entity capable of passing (or coming sufficiently close to passing, in most respects) the Turing Test.

We could then extend it to define degrees of intelligence - at least in certain domains - in similar terms: the more intelligent such an entity in this operational sense, the more it surpasses what a human could/would intelligently respond with in such a test, e.g., in terms of ability to correctly answer difficult problems in the domains of maths, logic, or coding.

The Turing Test has nothing to say about intelligence. It is purely about whether a programmed system can mimic the mere appearance of being so. The Eliza chatbot fooled many people, despite being a simple algorithm. People fell in love with it, even, which is disturbing. It's more a demonstration of how we can be fooled, deceived, manipulated, gaslight, cheated by tricksters and conmen who can pull off a convincing appearance.

It is why I am extremely wary of appearances ~ they can be most deceiving. There must be genuine confirmed substance.

Is syntax enough to produce reasoning? Please watch this section about attempts to mechanize a system to create and verify mathematical proofs:

https://youtu.be/ShusuVq32hc?t=1700

This question and its explorations are actually the inspirations for Godel's Incompleteness Theorems, which expose the fundamental flaws of mechanized "intelligence". There is simply no way for computational systems to have a sense of what is true, and without that foundation, there is no knowledge, information or understanding.

[Laird]

I've already watched and responded to that, @Valmar.