Richard Sutton's Bitter Lesson is one of the most cited essays in AI research — and it sits at the heart of one of the field's most heated debates. The essay argues that techniques which most effectively and scalably leverage compute always win in the long run. But Sutton goes further: he believes that large language models, despite their remarkable performance, are fundamentally not the right path to AGI. After sitting with his perspective more carefully, here's a fair attempt at steelmanning his position — and then honestly pushing back on it.
What Is Sutton's Bitter Lesson and Why Does It Matter?
The Bitter Lesson isn't simply saying "throw more compute at everything." That's the common misreading. What Sutton is actually arguing is that researchers should pursue techniques that scale gracefully with compute, rather than encoding human knowledge directly into systems. Historically, every time researchers tried to hard-code domain expertise — in chess, Go, speech recognition — the more general, compute-scalable approach eventually crushed it.
02:15
Explaining why compute spent on LLM deployment is wasted because the model isn't learning
Watch at 02:15 →
Applied to LLMs, Sutton's critique sharpens into something more specific. Consider how a large language model actually spends its compute: almost all of it is burned during deployment, running inference on billions of queries. And during that entire deployment period, the model is learning absolutely nothing. It's frozen. All learning happens in a special, separate phase called training — which is itself extraordinarily inefficient, requiring the equivalent of tens of thousands of years of human experience just to get a model to a useful level of capability.
Worse, that training data is fundamentally human-furnished. Even the reinforcement learning from verifiable rewards (RLVR) that powers modern reasoning models involves humans designing the environments, curating the tasks, and prescribing the skills. The agent isn't organically exploring the world. It's playing in a very carefully constructed human playground. Human data is an inelastic, hard-to-scale resource — and building your intelligence pipeline around it is, in Sutton's view, a scalability dead end.
Can LLMs Actually Achieve AGI or Are They a Dead End?
This is where the argument gets genuinely interesting. Sutton's position is that LLMs are building a model of what a human would say next, not a true world model of how the environment responds to actions. An LLM trained on all data up to 1900 probably couldn't derive special relativity from scratch — because relativity required breaking from the dominant human conceptual frameworks of the time. If your intelligence is anchored to human-derived concepts, there's a ceiling.
06:40
The fossil fuels analogy: pretraining data as a necessary but non-renewable intermediary resource
Watch at 06:40 →
But here's the pushback worth taking seriously: imitation learning and reinforcement learning are not mutually exclusive. Think about AlphaGo versus AlphaZero. AlphaGo was initialized on human game data; AlphaZero bootstrapped from scratch. AlphaZero was better — but AlphaGo was still superhuman. The human data wasn't actively poisoning the well. At sufficient scale it just stopped providing much marginal benefit.
Ilya Sutskever recently compared pretraining data to fossil fuels. The analogy has real reach. Fossil fuels aren't renewable, but that doesn't mean civilization made a catastrophic mistake by using them. You simply couldn't have gone from the water wheels of 1800 directly to solar panels and fusion reactors. You needed a messy, imperfect intermediary to get to the next stage. Pretraining on human data might be exactly that intermediary — not the destination, but an essential bridge.
After RLVR fine-tuning on pre-trained base models, we've seen these systems win gold medals at the International Math Olympiad and code entire working applications from scratch. These are ground-truth evaluations. The model either solved the unseen problem or it didn't. You couldn't have RLed a model to that level of performance starting from random weights — at least, we have no idea how to do that yet. The human prior was necessary to kick-start the RL process.
Is Imitation Learning Just Weak Reinforcement Learning?
One of the more provocative reframes here is treating imitation learning as short-horizon RL. Each episode is exactly one token long. The model makes a conjecture about the next token based on its world representation, and receives a reward signal proportional to prediction accuracy. That's RL — just with a very short horizon and a very specific reward signal.
11:20
AlphaGo vs AlphaZero — why human data helps but isn't strictly necessary at scale
Watch at 11:20 →
The standard objection is: "But it's not ground truth — it's just learning what a human was likely to say." Fair. But that's not the most important question. The more important question is: can we leverage imitation learning to help models learn better from ground truth? And the empirical answer at this point seems to be obviously yes. The base model provides the prior; RL provides the refinement toward real-world correctness.
Human cultural learning itself is more analogous to imitation learning than pure RL. We didn't invent the languages we speak, the legal systems we use, or the technologies in our pockets. Thousands — probably millions — of prior people built up that knowledge and passed it forward. We inherited it. That inheritance process looks a lot more like supervised learning than like an agent collecting scalar rewards from scratch. And yet it's been extraordinarily effective.
Do LLMs Have a Real World Model or Just Mimic Humans?
Sutton's position is that LLMs lack a true world model — specifically, a model of how the environment changes in response to different actions. What they have instead is a model of human language and human thought, which is a fundamentally different thing.
But there's a definitional problem lurking here. If we define "world model" by the process we think is necessary to build one (i.e., direct environmental interaction with action-consequence feedback), then LLMs fail by definition. But if we define it by capability — can the system reason flexibly and coherently about how the world works across biology, history, mathematics, code? — then the picture is much murkier.
LLMs are being asked to explain quantum mechanics, debug systems architectures, and reason through multi-step legal arguments — and they're doing it with remarkable coherence. The training process is actively incentivizing them to develop deep world representations, because predicting human language well requires understanding what humans are talking about. Whether we call that a "world model" or not starts to feel like a semantic debate rather than a substantive one.
What Is Continual Learning and Why Don't LLMs Have It?
Continual learning — the ability to update your model of the world on the fly as you encounter new information — is something every animal does naturally. An LLM undergoing RLVR learns on the order of one bit per episode, and a single episode might be tens of thousands of tokens long. The information extraction rate is abysmal compared to how biological agents learn from interacting with their environments.
In Richard Sutton's OaK (Observations and Knowledge) architecture, this is handled by what he calls the transition model — an inner learning system that's continuously updating based on observed tokens, driven by an outer-loop RL objective. The outer loop rewards are incentivizing the agent to extract maximum signal from every environmental interaction, not just from terminal reward signals.
For LLMs, the naive version of this — just fine-tuning on all observed tokens continuously — apparently doesn't work well in practice. But there might be cleverer approaches. One speculative idea: make supervised fine-tuning a tool call the model can invoke. The outer-loop RL then incentivizes the model to teach itself effectively using SFT in order to solve problems that exceed the context window. Models are already demonstrating something resembling continual learning within their context windows through in-context learning — and that capability emerged spontaneously from training incentives, not by design.
Why Is LLM Training So Shockingly Sample Inefficient?
A child learns to recognize a cat from a handful of examples. An LLM requires exposure to millions. The sample inefficiency of current training paradigms is one of Sutton's most pointed critiques — and it's hard to argue with. These models are trained on the equivalent of tens of thousands of years of human experience, and they still struggle with certain types of novel reasoning that a ten-year-old handles intuitively.
This inefficiency is partly a consequence of the architecture — transformers processing static datasets in a single frozen training phase — and partly a consequence of the learning signal. When your reward signal is "predict the next token," you're not extracting the rich causal and relational structure from experience the way a biological agent does through active exploration.
What Will AI Look Like After LLMs Are Obsolete?
Here's perhaps the most interesting synthesis: evolution does meta-RL to produce an RL agent. That agent can then selectively do imitation learning. With LLMs, we went the opposite direction — we first built a system optimized for pure imitation learning, and we're now hoping that sufficient RL on top of it produces a coherent agent with genuine goals and self-awareness.
Maybe that works. Maybe it doesn't. But even if LLMs do get to AGI first — which seems like the most probable near-term scenario — the successor systems those AGIs build will almost certainly look a lot more like Sutton's vision: continual learners, architecturally designed for efficient world modeling, not dependent on exhaustible human data pipelines.
Sutton's first-principles critique isn't just theoretical hand-waving. It's identifying real structural gaps — lack of continual learning, abysmal sample efficiency, dependence on human data — that are so pervasive in the current paradigm that most researchers have stopped noticing them. His decades-long perspective makes them obvious. Whether LLMs are the bridge to that future or the destination is the question the next five years will answer.
