Continual learning in AI refers to the ability of a model to update its knowledge and skills on the job — learning from real-world deployment the same way a new employee learns from experience, not just from a training manual. Right now, AI systems cannot do this in any meaningful way. Their weights are frozen after training, and no matter how many conversations they have or mistakes they make in production, none of that feeds back into making them smarter. That gap — between the vast tacit knowledge AI encounters during deployment and its inability to absorb any of it — is one of the most important unsolved problems in AI development today.
Can Reinforcement Learning Actually Get Us to AGI?
Here is the central bet every major AI lab is making right now: if you train an AI to accomplish millions of verifiable tasks across thousands of diverse reinforcement learning environments, you will have essentially built AGI. The reasoning is that this kind of training forges a genuine problem-solving agent — one capable of making progress on open-ended tasks for weeks on end, even in the face of errors, ambiguity, and dead ends.
02:15
The grindability argument: why AI needs replayable simulators to improve, not just verifiable tasks
Watch at 02:15 →
Optimists point to the LLM revolution as their proof of concept. Before large language models, natural language processing had a laundry list of hard, seemingly fundamental research problems. Then labs threw enough compute at the problem and those obstacles basically dissolved. The same logic, the argument goes, applies here. Yes, today's models are roughly one one-millionth as sample-efficient as humans during training. But training is a one-time cost, amortized across billions of sessions. What matters is how capable and general the model is during a session — and that has been clearly improving with more reinforcement learning.
The case for in-context learning as a substitute for continual learning is also compelling. People often say a new employee isn't net productive until six months in. But what if you could compress those six months into the context window? Architectural innovations are already dramatically expanding how much context a transformer can store, and there's no obvious reason that trend stops anytime soon.
But there's a telling quote from Anthropic's CEO Dario Amodei that hints at the limits of this approach. When asked about model performance degrading at long context lengths, he noted that models trained at short context lengths don't automatically generalize to serving at long context lengths. If short-horizon RL training doesn't straightforwardly generalize to long-horizon performance, it raises a deeper question: how are agents supposed to generalize from containerized white-collar tasks to navigating the real, messy, unverifiable world?
08:42
Why the KV cache can't substitute for real continual learning — and what the autistic-savant analogy reveals
Watch at 08:42 →
What Is Continual Learning in AI and Why Does It Matter?
Today, somewhere between 30 and 50 percent of a lab's compute goes to inference — to actually running the model in deployment. And right now, that compute plays zero productive role in improving the model. The model participates in countless conversations, gets exposed to rich organizational context, encounters its own real-world failure modes, and learns nothing from any of it. It's like having a genius intern who is banned from ever taking a real internship, no matter how many classroom case studies you throw at them.
This is the continual learning problem. And it matters because the most valuable training data doesn't exist in any datacenter. It exists in the organizations actually using these models. What are they using AI for? Where does it fail? What domain-specific tacit knowledge does it need to absorb? You can't pre-package that into a training run — by definition, it only gets revealed through real deployment.
Some might propose simply building up a larger and larger KV cache as the model interacts with more users. But that doesn't scale, and it's not how human intelligence works either. Human learning involves compression — distilling scattered experiences into refined intuitions, carved back into the underlying parameters. Interestingly, there are autistic savants who can recall random tables of numbers with near-perfect fidelity years later, essentially what a model does in context. And that sheer volume of recall tends to impair their ability to work with abstractions and metaphors. Human learning is less about perfect recall and more about chiseling the right big-picture knowledge into the weights.
Why Is AI Computer Use So Much Slower Than Coding or Math?
Computer use seems like it should be one of the most trainable AI skills. Tasks like "did the Etsy order get delivered" or "is the event venue booked" are clearly verifiable. So why has progress on computer use lagged so far behind domains like coding and math?
14:30
On-policy self-distillation explained: how the veteran model teaches the base model without a verifiable reward
Watch at 14:30 →
Part of the answer is less multimodal training data. But there's a more underrated reason that reveals something deep about the limits of current AI training: a domain doesn't just need to be verifiable — it also needs to be grindable. Grindable means you can run thousands of parallel rollouts against a deterministic, replayable simulator, starting from the same initial state every time.
For coding, this is easy. Spin up a thousand identical containers with a software repo missing a feature, send a thousand parallel agents at it, measure the results. Clean, scalable, repeatable. But you can't run a thousand agents through Amazon's checkout flow to train computer use — Amazon will detect the bots and shut it down. You'd have to build high-fidelity clones of Slack, Gmail, and every other common application. That's currently a massive, labor-intensive undertaking.
The good news is that as AI gets better at coding, it will get better at building those application clones with high fidelity — and training against them serves as a great RL objective for coding in its own right. But the deeper lesson stands: without a replayable simulator, AI training stalls. And for the vast majority of valuable real-world skills — building a business, winning court cases, trading profitably, winning elections — you simply cannot build a deterministic simulator inside a datacenter.
What Is RLVR and Why Are AI Labs Betting Everything on It?
RLVR stands for Reinforcement Learning from Verifiable Rewards. The core idea is to train AI on tasks where correctness can be objectively checked — math problems, coding challenges, logic puzzles — and use that verification signal to improve the model through reinforcement learning. Unlike human feedback, which is subjective and inconsistent, verifiable rewards are clean and scalable.
The labs' bet is that RLVR generalizes. That training on enough containerized, reproducible environments will produce a broadly capable agent — one that can make plans, execute them, iterate on failure, and rapidly pick up new skills within a session. The ultimate stress test of this idea: if you dropped a sufficiently RLVR-trained AI into Texas politics in 1948, could it give better advice than Lyndon Johnson on winning a Senate seat? If you gave it $100 million in 2002, could it build SpaceX? Whether RLVR can generalize that far is an open empirical question — and it depends heavily on how much compute gets thrown at it.
What Is On-Policy Self-Distillation and How Does It Work?
On-policy self-distillation (OPSD) is one of the most promising current ideas for solving the continual learning problem. The core mechanism: take a base model and a "veteran" version of that model that has accumulated experience across a long real-world session. Then train the base model to make the same predictions the veteran model would make when solving actual problems.
OPSD has two key advantages over naive reinforcement learning for continual learning. First, it doesn't require an outer-loop verifiable reward — you just need a model capable of learning the right things in context, and then you distill that back into the weights. Second, it provides a much denser supervision signal. Instead of propagating a single reward through an entire trajectory, you train on the per-token probability difference between teacher and student at every step.
It's also superior to simple supervised fine-tuning for this use case. Naively training a base model to predict every token from a session transcript is like trying to improve at your job by memorizing a transcript of every single workday. What actually makes you better is consolidating the handful of insights that actually matter. RLVR-style updates are naturally sparse — they only change the model as much as necessary to achieve the right outcome, which is exactly the property you want for continual learning. You update the minimum required and avoid overwriting everything else the model already knows.
What Is 'Dreaming' or Test-Time Training in AI Models?
Beyond OPSD, there's a more speculative but potentially transformative idea sometimes called dreaming or test-time training. The concept: instead of just compressing context into a brief summary when a session gets too long, the model spends significant compute building and training against its own simulated version of what it is witnessing in the real world.
Think of EfficientZero, the DeepMind follow-up model that could match a novice human at an unseen Atari game after just two hours of play — because for each real-world step, it was running dozens of simulated games internally. Future LLMs might do something analogous: consume far less real-world data while rehearsing endlessly against internal environments they construct themselves.
The gap, of course, is that simulating a game of Go is far simpler than simulating the real world. But if this capability develops, it would represent a fourth axis of scaling alongside pretraining, RL, and inference-time compute. Instead of hitting /compact and getting a lightweight summary, you'd hit /dream — incinerating large amounts of compute to build a video-game version of the task at hand and train against it before proceeding.
How Could AI Learn From Real-World Deployment by 2027?
Here's one plausible scenario for how continual learning arrives. All of today's RLVR training produces an agent that, when thrown at an unfamiliar problem, can get its bearings, try strategies, and iterate past roadblocks. That's the crucial foundation. Once you have that, you send it into real-world work — even tasks off the training distribution.
Suppose effective context windows expand to where an AI can work alongside you for a full week of wall-clock time. At the end of that week, you give it a review — thumbs up or thumbs down. A thumbs up triggers a distillation process: the base model absorbs what the session-model learned, using OPSD, dreaming, or some combination of techniques not yet invented. The AI gets better at domains adjacent to what it was originally trained on. In the next round, it gets better at domains adjacent to those. The frontier of AI capability expands far beyond what any pre-deployment training run could have explicitly covered.
The implication is profound. The main driver of AI improvement would no longer be what happens before a model ships. It would be what happens after — every interaction, every organization, every deployment adding to a collective intelligence that grows smarter with every session. That's a very different world from the one we're in today, and getting there requires solving the continual learning problem. The tools — OPSD, dreaming, longer context, denser RL signals — are beginning to take shape. The question is whether they converge fast enough to matter.
