The Dark Side of Large Context Windows
- Large context windows can lead to decreased performance and increased errors
- Recursive calls can help control token use, but have limitations
- Some users have found success with smaller context windows and external symbolic recursion
The Buzz Score
The Internet’s Verdict: 60% Cautious, 40% Optimistic
Forum Voices Weigh In
Some users have reported issues with large context windows, including decreased performance and increased errors.
I’ve been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user’s top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller.
Others have found success with smaller context windows and external symbolic recursion.
Considerations about what goes on in agents internally will probably not be part of software development for long. Personally, I already see LLMs and agents as blackboxes. I give each feature request to multiple LLMs and then compare the results.
Despite these concerns, some users have reported positive experiences with large context windows.
This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don’t see this problem.
Focus Keyword: Context Windows
