Large language models have a context window problem, and it hasn't really been solved. Despite dramatic leaps in model quality, reasoning ability, and parameter counts over the last few years, most LLMs are still capped at 2,048 to 4,096 tokens — and only a handful push beyond 16K. If you've ever tried to feed a research paper, a long codebase, or a big email thread into an LLM and hit that wall, you already know how frustrating this is. So why do large language models have such limited context windows, and is there any real fix on the horizon?
Why Do LLMs Have Such Limited Context Windows?
It comes down to three compounding problems that all trace back to how attention works in transformer-based models. Attention is what lets a model understand relationships between tokens — but it doesn't scale gracefully. As context length grows, the cost of computing attention explodes. Here are the three core issues:
02:15
Visual showing a 2048-token example and how quickly context fills up on real tasks
Watch at 02:15 →
- GPU memory limits: This is a hard yes-or-no constraint. Either the attention computation fits in your available GPU memory, or it doesn't. As context grows, memory requirements scale sharply — a 5,000-token context on a model like MPT-30B can already saturate a full 80GB H100 GPU at 16-bit precision. That's just 5K tokens, not even a third of that model's claimed 16K limit.
- Exponentially increasing processing time: Even if you have the memory, the time cost is punishing. Doubling the context doesn't double the compute — it multiplies it. On an H100, generating a couple of short sentences at a 16K context can take well over a minute. That makes it impractical for training and nearly useless for production inference.
- Quality degradation: Even if you somehow clear the memory and time hurdles, the model's actual comprehension suffers badly as context grows. More on this in a moment — because this third problem is arguably the worst of the three.
So while the numbers on paper — 16K, 65K, even 84K tokens — can sound impressive, those figures often don't tell you anything about whether the model can actually use that context meaningfully.
How Much GPU Memory Does a Large Context Window Need?
The numbers here are humbling. Running MPT-30B at even a modest 5,000-token context window will push an 80GB H100 SXM to its limit at 16-bit. You can run at 8-bit or even 4-bit quantization to claw back some breathing room, but that only gets you so far — maybe a rough halving or quartering of memory usage. The exponential scaling of attention means you're always playing catch-up.
Jensen Huang can only manufacture new GPUs so fast, and no realistic hardware roadmap solves this problem on its own. Even an 8x H100 SXM build — about as powerful a consumer-accessible AI rig as you can put together — is the hardware benchmark Microsoft uses when describing LongNet's theoretical 1-billion-token capacity. That should tell you something about how extreme this challenge really is.
07:40
Stanford research showing the U-shaped quality curve across multiple LLM models
Watch at 07:40 →
Can LLMs Actually Read an Entire Research Paper?
Short answer: barely, and not well. A typical research paper runs anywhere from 6,000 to 12,000 tokens or more. A 2K context window won't get you through the introduction. Even a 4K window often can't hold a full paper. At 5K, you're getting closer — but as the memory discussion above shows, that already strains serious hardware.
In practice, feeding the full LongNet paper (roughly 5,000 tokens) into MPT-30B at a 5K context was possible, but barely. The model did seem to perform better on topic-specific questions when the paper was in context versus relying on the base model's knowledge alone — suggesting that large contexts can function as a kind of short-term memory injection. But the hardware overhead was enormous for a relatively modest result, and the quality still left something to be desired.
Techniques like chunking, summarizing, and vectorizing text exist as workarounds, but none of them truly replicate having everything available in a single coherent context. They're patches, not solutions.
11:05
LongNet perplexity comparison between dilated attention and sparse transformer baseline
Watch at 11:05 →
What Is the 'Lost in the Middle' Problem in LLMs?
Even when a model technically supports a large context window and can process it within memory and time constraints, there's a third, nastier problem: the model doesn't pay equal attention to all parts of the context. Research from Stanford found that information placed in the middle of a long context is remembered significantly worse than information at the beginning or end.
The result is a U-shaped quality curve. Tokens near the start of context? Great recall. Tokens near the end? Also decent. Tokens in the middle? Often effectively lost. And critically, this isn't a bug in any one model — it showed up across every model tested, suggesting it's a fundamental property of how attention works, not something a better fine-tune or a smarter architecture tweak will fix easily.
This means that even if you solve the memory problem and even if you solve the speed problem, you still have a model that's unreliable about what it actually remembers from a long input. That's a serious limitation if your goal is, say, parsing a 50-page document or maintaining coherence across a lengthy conversation.
How Does Microsoft LongNet Scale to 1 Billion Tokens?
This is where things get genuinely interesting. Microsoft Research's LongNet paper makes a bold claim: scaling transformer context lengths up to one billion tokens — potentially enough to hold the entire internet in a single context window. It's a dramatic headline, and it's worth understanding both what it actually proposes and where the skepticism is warranted.
LongNet's core innovation is called dilated attention. Rather than computing full attention across every token in context — which is what makes traditional attention so expensive — LongNet splits the context into segments and applies attention sparsely within and across those segments. These segment-level attention calculations run in parallel and are then combined into a final output.
The key insight is that you can tune the trade-off between coverage and compute by adjusting two parameters: segment size and sparsity. Want to cram a billion tokens into a reasonable memory footprint? Increase sparsity. Want better coverage at the cost of more compute? Tighten it. This flexibility is what lets Microsoft claim LongNet could theoretically fit into something like an eight-GPU H100 build while remaining computationally comparable in speed to a standard transformer running at 8K–16K context.
In their benchmarks, LongNet's dilated attention showed modestly better perplexity scores than a sparse transformer baseline across tested context lengths, which is a promising sign. But the paper only compared up to 32,000 tokens — a long way from a billion.
What Is Dilated Attention and How Does It Work?
Dilated attention is conceptually similar to dilated convolutions in computer vision — instead of looking at every adjacent element, you look at every nth element, spreading your receptive field without increasing compute proportionally. In LongNet's case, this means attention isn't computed between every pair of tokens. Instead, the context is segmented, attention is applied within those segments at varying levels of sparsity, and the results are aggregated.
Think of it like this: if you want to fit a billion tokens and you compute attention every millionth token, you can technically do it — but you're necessarily losing a massive amount of information along the way. The question is whether the hierarchical combination of sparse attention across segments can recover enough of that information to remain useful. LongNet argues yes, at least in terms of perplexity. Whether that holds at truly massive scales remains unproven.
Does LongNet Actually Solve the Quality Problem?
Here's the honest answer: probably not fully, at least not yet. Even in Microsoft's own benchmarks, dilated attention shows quality degradation as context scales — the perplexity scores worsen as you go from 8K to 32K tokens, at a rate that looks concerningly similar to standard attention. If quality is already visibly declining between 8K and 32K, extrapolating that curve to one million or one billion tokens raises serious questions.
The lost-in-the-middle problem also isn't directly addressed by the dilated attention mechanism. Segmenting and sparsifying context might actually make middle-of-context forgetting worse in some configurations, not better, depending on where segment boundaries fall.
That said, LongNet is genuinely important as a step forward. It credibly addresses the memory problem and the processing time problem in ways that previous approaches haven't. Those are two out of three, and that's not nothing. The quality question is the remaining frontier, and it's the hardest one.
One thing is certain: standard attention has to change before we'll see meaningful progress on context windows for transformer-based LLMs. LongNet is one serious attempt at that change. Whether it — or something inspired by it — eventually cracks the quality problem too is the real question worth watching.
