Back to all posts

Bigger Context Windows Are Making Your Agents Worse

Elie Bitar · 2026-06-08 · 3 min read

Server racks in a data center, illustrating AI infrastructure and LLM serving

Everyone in the AI space is celebrating context window expansion. Gemini handles 1M tokens. Claude supports 200K. GPT-5 pushes 400K. The benchmark race is in full swing and the narrative is straightforward: bigger context = smarter agents.

I disagree. And I have a few months of burned agent runs to back it up.

The Symptom Nobody Talks About

There is a name for what happens when you stuff too much into an LLM's context: context rot. It is when a long-running agent starts degrading mid-task — forgetting earlier instructions, repeating tool calls it already made, losing the thread of what it was originally asked to do.

I first noticed it clearly while building Hermes, my personal AI agent framework. I had an agent that worked great on short tasks, but on anything that ran more than 10-15 tool calls deep, it would start going sideways. It would re-fetch a file it had already read. It would produce a summary that contradicted an answer it gave 20 messages earlier. Classic context rot.

My first instinct was: "I need a bigger model, or a longer context window." That instinct was wrong.

What the Research Actually Says

Amazon published a paper with a blunt finding: "Context length alone hurts LLM performance despite perfect retrieval." Even when the relevant information is guaranteed to be in context, adding irrelevant surrounding content degrades accuracy. The model is not a perfect scanner — it is a statistical pattern matcher, and noise drowns signal even when the signal is physically present.

This is not a fringe finding. It aligns with the "lost in the middle" problem documented by Stanford and others: LLMs are worse at using information in the middle of a long context compared to the beginning or end.

For agents, this compounds badly. Every tool call adds output to context. Every reasoning step adds more. By turn 20 of a complex task, your agent is operating on a context that includes failed attempts, redundant observations, and obsolete intermediate states. It does not know to ignore any of that.

What Actually Fixed It

I made one architectural change in Hermes: instead of passing the full conversation history + all tool outputs into every inference call, I built a rolling compaction step. Every 8 turns, the agent summarizes what it has learned so far into a compact state block, then continues with only that summary plus the last 3 turns of full context.

The result was immediate and measurable:

  • Tasks that previously degraded around turn 15-20 now run clean to turn 40+
  • The agent stopped repeating tool calls it had already made
  • The summaries forced the agent to consolidate its understanding rather than just accumulating raw output

No hardware changes. No model upgrade. Just less context.

For anyone running on vLLM in production: the --max-model-len flag is not just a memory dial. It is a quality dial too. If you are running 32K context capable models but your typical agent task only needs 8K, capping the max context forces you to be intentional about what goes in. That intentionality pays off.

The Deeper Problem

The industry is building toward agents that can hold entire codebases in context, entire conversation histories, entire document sets. That capability is real and sometimes genuinely useful — analyzing a full repo in one pass, for instance.

But there is a category error happening. Teams are assuming that if the context window is big enough, they do not need to think about context management. They are treating context as "just memory" rather than as the primary signal-to-noise problem.

The teams building reliable agents are not the ones with the most context. They are the ones with the most intentional context. There is a meaningful difference between "I gave the agent everything" and "I gave the agent exactly what it needs at each step."

Summarize, do not stuff. Build compaction into your agent loop from the start. Treat context size as a quality metric, not just a resource metric.

The Bottom Line

If your agents are getting flaky on long tasks, the answer probably is not a model with a bigger context window. The answer is probably to stop putting so much junk in the context you already have.

If you are building agents on self-hosted models and want to compare notes on context management strategies, connect with me on LinkedIn — I'm curious what compaction patterns other practitioners are using.


Working on something similar?

If this resonated with a workload you're trying to integrate AI into, send me a note. Happy to compare notes or help directly.

Get in touch
An unhandled error has occurred. Reload 🗙