AI Agent Context Management: Budgeting, Compaction, and Retrieval

AI agent context management is the practice of deciding what goes into a model's context window on every step of a run: which instructions, which tool results, which retrieved documents, and which prior turns actually earn a place, and what gets summarized, dropped, or fetched later instead. It matters because an agent doesn't get one shot at a prompt, it loops, and every pass through that loop competes for the same limited, increasingly crowded space. Treat context as free scratch space and an agent gets slower, more expensive, and less accurate the longer a task runs. Treat it as a budget and it stays sharp.

Why This Is an Agent Problem, Not a Chat Problem

A single question to a chatbot fills a small, predictable slice of a context window: a system prompt, the question, an answer. An agent is different. How AI agents work describes the loop it runs, perceive, reason, act, observe, repeat, and every one of those passes can add more to what the model has to hold in mind. A task that takes ten steps has read ten batches of tool output, made ten decisions, and possibly retrieved several documents by the time it finishes. None of that disappears automatically. It sits in the context window unless something deliberately manages it.

Anthropic's engineering team frames the underlying problem well in its guide to context engineering for AI agents: context is a finite resource with diminishing marginal returns. Every additional token past what a step actually needs doesn't just cost money, it can actively hurt performance, because a transformer's attention has to spread across everything in the window, and it spreads more thinly as that window fills. That's a different failure mode than running out of room. It's degrading quietly while there's still room left, a dynamic Chroma's research on long-context performance documented directly, testing 18 leading models and finding accuracy can start dropping well before a window is technically full. AI Agent Memory covers that research in more depth from the storage side; this page picks up where it leaves off, on what to actually do about it.

What Actually Fills Up an Agent's Context

Before you can budget a context window, you need to know what's competing for space inside it. On a typical agent run, five categories fight for the same tokens:

Tool call results are usually the biggest offender in practice. How AI agents use tools covers how a tool call returns a raw result the model reads before deciding what's next, and a lot of APIs return far more than an agent needs: a full customer record when it needed one field, a full search response when it needed the top three results. The AI Research Agent shows what happens when this goes unmanaged: a brief that pulls ten full sources into context instead of three focused excerpts isn't just slower to generate, it's diluting the model's attention across material that mostly won't make it into the final synthesis anyway.

Budgeting: Decide What Earns a Place Before It's Full

A context budget is a simple idea applied deliberately instead of by accident: decide, before a run starts, roughly how much of the window each category above is allowed to consume, and enforce it as the run goes. Anthropic's framing for this is to look for the smallest possible set of high-signal tokens that gets the model to the right outcome, not the largest set that might be relevant.

In practice, that means a few concrete habits:

• Trim tool results before they enter context, not after. Extract the two or three fields a step actually needs from an API response instead of passing the whole payload forward.
• Keep the system prompt tight. A system prompt padded with every edge case anyone's ever hit is a permanent tax on every single run, whether that edge case shows up or not.
• Cap history, don't let it run open-ended. Decide how many prior steps of a long-running task stay in full detail versus getting compacted (more on that below).
• Scope retrieval narrowly. Pull the handful of chunks a question actually needs instead of a whole document, a lesson RAG for AI agents covers from the grounding side; it's also a context-budget decision, not just an accuracy one.

Compaction: Summarize Before You Overflow, Not After

Compaction is what you do when a run has already accumulated more history than it needs to keep in full: summarize what's happened so far into a compressed form, then continue the task from that summary instead of the entire transcript. Anthropic's guidance is specific about how to do this well: aim for high recall first, make sure the compaction step captures every piece of information the next step might actually need, then tighten for precision once you've confirmed nothing important gets dropped. Getting that order backwards, optimizing for a short summary before confirming it's a complete one, is how a compacted agent quietly starts forgetting things that mattered.

A related, older idea worth knowing is MemGPT, a proposal that borrows tiered memory from operating systems: keep a small amount of "hot" information in the model's immediate context, and page everything else out to slower, larger storage, pulling it back in only when a step actually needs it. Compaction and just-in-time retrieval, covered next, are both practical versions of that same paging idea.

Just-in-Time Retrieval Beats Frontloading Everything

The instinct when an agent might need a piece of information is to load it into context up front, just in case. The better pattern, and the one Anthropic's guidance leans on heavily, is just-in-time retrieval: give the agent a lightweight reference, a file path, a record ID, a search tool, and let it pull the actual content only at the step where it's needed. That mirrors how a person works a task. Nobody memorizes an entire policy manual before answering one question about parental leave; they look up the relevant section when the question comes in.

This is exactly the mechanic RAG for AI agents describes: retrieval, inside an agent, is really a tool call like any other, sitting in the perceive step of the loop, triggered when a specific step needs specific information rather than loaded wholesale at the start. The context-management payoff is separate from the grounding payoff. Even if a model could technically fit an entire knowledge base into its window, doing so would still be the wrong move, because every token in there past what the current step needs dilutes the model's attention on the tokens that actually matter right now.

Offloading to Sub-Agents: A Clean Window for a Focused Job

When a single step inside a larger task needs to do a lot of reading, research, or exploration to accomplish something narrow, one of the more effective context-management moves is handing that step to a separate sub-agent instead of doing it inline. The sub-agent gets its own clean context window, does the heavy work there, and returns a condensed result, often in the range of 1,000 to 2,000 tokens per Anthropic's own examples, back to the main agent instead of dragging its entire working process back with it.

Multi-agent systems covers the orchestration side of this pattern in depth: when splitting a job across agents earns its complexity versus when it's over-engineering a job a single agent could still handle. From a pure context-management lens, the case for a sub-agent is narrower and more mechanical: a step that would otherwise flood the main agent's window with exploratory detail it doesn't need to carry forward is a good candidate to isolate, summarize, and hand back.

Watching Context Health After You Ship

A context strategy that works in testing can still drift once real traffic hits it: tool outputs get verbose after an API update, a knowledge base grows, an edge case gets hardcoded into the system prompt and never gets removed. AI agent observability already names the metrics worth tracking here: rising loop iterations per run and rising cost per completed task are both early signs that context is bloating quietly, often before accuracy visibly drops. AI agent cost optimization covers the caching side of the same coin, making reused context dramatically cheaper to resend, which matters alongside compaction rather than instead of it, since even well-cached context still has to be attended to on every call.

If you're evaluating platforms that handle large codebases or long-running sessions well, our developer tools comparisons and the AI coding assistant buying guide both cover context-window handling as a real differentiator, not a footnote, since it's one of the clearest places a coding assistant's quality shows up on a large project.

Key Facts

• Anthropic frames context as a finite resource with diminishing marginal returns: extra tokens past what a step needs don't just cost more, they can dilute the model's attention on what actually matters.
• Tool call results, not conversation history, are usually the fastest-growing part of an agent's context, since many APIs return far more than a single step needs.
• Compaction should optimize for recall first (capture everything that might matter) and precision second (tighten the summary), not the other way around.
• Sub-agents handling a narrow, exploration-heavy step typically return a condensed result of roughly 1,000 to 2,000 tokens to the main agent instead of carrying its full working context forward.
• Rising loop iterations and rising cost per completed task are early warning signs of context bloat, often visible before accuracy drops.

Where to Go Next

Context management is the day-to-day discipline that keeps an agent fast, accurate, and affordable as tasks get longer. AI Agent Memory covers the storage side in more depth, RAG for AI agents covers retrieval as a grounding technique specifically, and AI agent observability covers how to catch context bloat after launch, before it shows up as a cost spike or a quiet drop in accuracy.