Managing Context

Overview

Every loop accumulates history. Each iteration adds the tool call it made, the result it got, and the reasoning that connected them, and all of it travels with every subsequent request.

Context is a budget with a hard ceiling, and four things compete for it. Understanding the competition explains a failure mode that otherwise looks inexplicable.

Four competing claims

graph LR
  A["Instructions<br/><i>fixed, mandatory</i>"] --> W["Context<br/>window"]
  B["Tool schemas<br/><i>grows with<br/>tool count</i>"] --> W
  C["Retrieved material<br/><i>what makes answers<br/>correct</i>"] --> W
  D["Loop history<br/><i>grows every<br/>iteration</i>"] --> W

  classDef fixed fill:#7C7C7C,stroke:#5E5E5E,stroke-width:2px,color:#ffffff
  classDef grows fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef squeezed fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  classDef win fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  class A fixed
  class B,D grows
  class C squeezed
  class W win

Four claims on one budget. The terracotta one is what makes answers correct, and it is the one that gets squeezed — because it is the only claim with no natural floor.

The failure mode this explains: a late step in a long interaction can be worse-informed than an early one. The loop began with ample retrieved material and good instructions; twenty iterations later, its own history has consumed most of the window, and the material it needs to answer correctly has been pushed out or was never re-fetched. Quality degrades during a single interaction, which no per-request metric will show you.

Three mechanisms, three different losses

Prune. Clear stale tool results and completed reasoning from the transcript. The structure of the conversation stays; the detail is deleted. What it loses: the detail, permanently. If a later step needs that result, it must fetch it again. Cheap, predictable, and the right default for long loops where old tool output has served its purpose.

Summarise. Replace earlier history with a generated summary when approaching the limit. What it loses: fidelity, silently. And there is a subtler problem: the summary is itself model-generated, which makes it another probabilistic artifact — and from this point on it is what the system believes happened. An error in the summary is now load-bearing and invisible, since the original is gone. Use it when the alternative is exceeding the window, not as routine housekeeping.

Externalise. Write state to a store and read it back when needed. What it loses: nothing from the window — but it gains you a store with its own obligations. Never put secrets in it. Scope it per tenant. And treat what comes back as untrusted on the way in, because the thing that wrote it was the model.

A fourth technique sidesteps the problem rather than managing it: compose a sequence of tool calls in code so intermediate results flow between steps without ever entering the model’s context. Only the final output returns. Where a sequence produces large intermediates nobody needs afterwards — reading twelve records to compute one total — this is strictly better than pruning them afterwards, because they never arrive.

Choosing

The decision is not “which mechanism” but what must survive:

What must survive	Mechanism
Nothing — the detail has served its purpose	Prune
The gist, because the window is about to be exceeded	Summarise, reluctantly
Across sessions, or beyond this interaction	Externalise
Exactly, with no loss — an action log, an audit record	None of the above. Keep it outside the context entirely

That last row is the important one. Anything requiring exactness — what actions were taken, with what arguments, on whose behalf — belongs in your own storage, not in the model’s context, and certainly not in a summary of it. A context window is a working surface, not a record.

The cost of managing context at all

Each mechanism adds a component, and each introduces a failure mode of its own. Pruning can remove something a later step needed, producing a loop that re-fetches the same thing repeatedly — which looks like wandering and is actually amnesia. Summarising can quietly corrupt the system’s understanding. Externalising adds a store that can be unavailable, stale, or leak across tenants.

The cheapest form of context management remains the one from Section 1: a tighter bound on the loop. A loop capped at a handful of steps rarely needs any of this. Reach for these mechanisms when the bound is genuinely insufficient, not before.

What must survive the action loop

The action path is capped at 12 tool calls, which keeps the context problem modest by design. Even so, the question has a clear answer.

Must survive exactly, and therefore lives outside the context: the plan as confirmed, every action attempted with its arguments and outcome, the tenant scope, the idempotency keys issued, and which bound terminated the loop if one did. All of that is the audit record Lena Fischer will be shown and the trace a support engineer will read later — it cannot be a summary, and it cannot be something the model might paraphrase.

Can be pruned: raw read results once they have been used. A full job record fetched at step two, having yielded the two fields the plan needed, is dead weight by step eight.

Should not be summarised: anything in the first category. An action log that says “issued a credit of around forty dollars” is not an action log. This is the clearest case in the module where the convenient mechanism is the wrong one.

Externalised, carefully: anything the system should remember across conversations with the same customer. That store inherits every rule from Module 4’s data strategy — per-tenant isolation, no personal data beyond what is necessary, and inclusion in the erasure path, because a right-to-erasure request reaches it exactly as it reaches the vector index.

And the measurement that would change the design: if the ticket-search results consume enough of the window that later steps in the same interaction degrade, that is the trigger for the read-heavy worker from the first topic of this section. The worker exists to protect this budget, which is why these two topics belong together.

Feeds the workbook

Your agent design should state what must survive exactly and therefore lives outside the context. Your observability plan depends on it: a trace assembled from the model’s working context is not evidence.

→ Context is budgeted and bounded. Now the integration question: every tool is a connection, and connections sprawl. — MCP as a seam