Context Overflow and Silent Truncation

Overview

The previous antipattern compounds. This one is quieter: it produces no error, no warning, and no signal of any kind.

Content exceeds the window, something gets dropped, and the system answers anyway — fluently, confidently, and without the part that mattered.

What actually happens

Module 5 established the context budget: instructions, tool schemas, retrieved material and accumulated loop history, competing for a fixed ceiling. Two of those grow during an interaction.

When the total exceeds the ceiling, something has to go. Depending on the layer that handles it, that might be the oldest turns, the middle of a long document, the end of a tool result, or the retrieved material that arrived last. Sometimes a request is rejected outright, which is the good case because at least it is visible. Often it is quietly trimmed.

The output is then produced from a partial view, and nothing in the response indicates that. This is the purest form of Module 1’s quiet failure: a correct-looking answer derived from incomplete input.

graph LR
  A["Instructions<br/><i>fixed</i>"] --> W["Window<br/>ceiling"]
  B["Tool schemas<br/><i>fixed-ish</i>"] --> W
  C["Retrieved material<br/><i>grows</i>"] --> W
  D["Loop history<br/><i>grows</i>"] --> W
  W --> E["<b>Dropped silently</b><br/>no error · no flag ·<br/>answer still produced"]

  classDef fixed fill:#7C7C7C,stroke:#5E5E5E,stroke-width:2px,color:#ffffff
  classDef grows fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef ceil fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef gone fill:#D8D3CB,stroke:#B8B2A8,stroke-width:2px,color:#3F3F3F
  class A,B fixed
  class C,D grows
  class W ceil
  class E gone

Four claims on a fixed ceiling, two of them growing. The drained box is drawn drained deliberately — the thing that happens leaves no trace.

Why it is hard to spot

Three reasons it survives in production longer than it should.

The symptom looks like a model problem. An answer that ignores a document which was supplied reads as the model being careless. The actual cause is that the document was not there by the time the model saw the request.

The boundary moves. A request shape that worked yesterday overflows today because the conversation is two turns longer, the retrieved document is larger, or a tool returned more than usual. There is no change to blame, which makes it look intermittent and therefore like flakiness.

Nothing in the response is wrong-looking. Module 1’s point: output is fluent by construction. A partial view produces a complete-sounding answer.

Detection

Count tokens before sending, as a gate. Not an afterthought — a check with a defined behaviour when it fails. Measuring input size is cheap and exact, and it is the only way to know you are near the edge rather than over it.

Record actual context usage per request, and alert on approach rather than on breach. Module 5’s trace should carry it; if it does not, that is a one-line addition with a large payoff.

Watch for the specific symptom: an answer that contradicts or ignores material that was supposed to be in the prompt. That is close to diagnostic, and it is worth a named check in your eval set — Module 6’s both-directions discipline applied to a failure mode rather than a behaviour.

The structural fixes

Budget the window with headroom, treating the largest plausible case rather than the typical one. Module 6’s “price the tail” instinct, applied to tokens.

Gate user-supplied input by size. Count first, then truncate deliberately, summarise deliberately, or route it elsewhere. A deliberate truncation you chose is a different thing from one an implementation detail performed for you.

Bound the loop, which bounds history — Module 5’s cap again, earning its keep for a third reason.

Manage the budget explicitly with the three mechanisms from Module 5, remembering Module 6’s correction that pruning is a window tool and not a cost saving.

And the rule that matters most: fail loudly rather than truncate silently. If a request does not fit, that is a condition with a designed response — a degradation rung from the previous section, a handoff, a narrower retrieval — not something to quietly clip. The whole antipattern is a missing error condition.

Where the ceiling gets reached

Two paths, two different pressures.

The answer path is short — retrieve, generate, check — so history barely grows. Its pressure comes from retrieved material: Module 4’s retrieve-small-supply-large decision expands each matched chunk to its enclosing section, and a question matching several sections of long documentation can assemble a large context from an innocuous-looking query.

The action path is where it bites. Twelve steps, each appending a tool call and its result, and some of those results are full job records. By step ten the history is substantial — and this is the path where a dropped detail has consequences, because the plan presented for confirmation may be built on a job record that was trimmed away.

The gate: count input tokens before every model call. If the request exceeds the budget with headroom, do not send it — reduce retrieval depth first, then prune the oldest tool results deliberately, and if it still does not fit, hand off with the context gathered so far. That last branch is rung four of the degradation ladder, and overflow is now one of its named entry conditions, which it was not in the Module 3 design.

The measurement: context usage recorded per request in the Module 5 trace, reported as a distribution rather than an average, with an alert at 80% of the ceiling. The distribution matters because the mean will look comfortable for a long time while the tail is already clipping.

The eval case: a question whose correct answer depends on material deliberately placed late in a long context, included in the 300-question set. If the system answers it correctly, nothing is being dropped at that size. It is the cheapest possible regression test for this failure, and it costs one case.

Feeds the workbook

Add this to your antipattern review. The detection question to answer honestly: if this were happening in your design today, what would tell you? If the answer is nothing, that is the finding.

Check your understanding

1. Which path in your design is closest to its context ceiling, and what measurement would tell you?
2. A request does not fit. Write the behaviour — what is reduced, in what order, and when do you stop trying?
3. Design one eval case that would detect silent truncation. Why does it work?

---

→ Both failures so far are things that happen. The next three are things that were never built. — Antipatterns of omission