Bounding the Loop

Overview

Every pattern in the previous topic can, in principle, run forever. In practice it runs until something stops it, and the quality of your system depends on whether that something was designed or accidental.

This topic is about the bounds, and about a distinction that matters more than it first appears: some bounds tell the system to wrap up, and others simply cut it off.

Four bounds, doing four different jobs

graph LR
  A["<b>Step cap</b><br/>max iterations<br/><i>enforced, blunt</i>"] --> B["<b>Token or task budget</b><br/>the loop knows and paces itself<br/><i>advisory, graceful</i>"]
  B --> C["<b>Wall-clock timeout</b><br/>the user's patience<br/><i>enforced</i>"]
  C --> D["<b>Spend cap</b><br/>per interaction and per month<br/><i>enforced, final</i>"]

  classDef hard fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  classDef soft fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef mid fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  class A hard
  class B soft
  class C mid
  class D hard

Four bounds. The olive one is the only one the system can respond to; the others simply fire. A design with only terracotta bounds truncates mid-thought every time it hits a limit.

A step cap limits iterations. Simple, reliable, and blunt: when it fires, the loop stops wherever it happens to be.

A token or task budget is the interesting one. Here the loop is told how much it has and can pace itself — consolidating work, skipping optional verification, and producing a usable partial result before it runs out. This is categorically different from a cap, and the difference shows up in output quality at the limit.

A wall-clock timeout represents the user’s patience and the latency budget from Module 2. It is enforced and it is independent of the other two: a loop can be well within its step and token budgets and still have taken far too long.

A spend cap exists at two levels — per interaction, protecting the p95 cost ceiling, and per month, protecting the budget somebody approved. The monthly one should be enforced by something outside the application, because the failure it guards against is the application behaving unexpectedly.

Advisory and enforced are both required

The distinction deserves stating plainly:

An advisory bound changes how the system behaves as it approaches the limit. An enforced bound guarantees the limit is not exceeded.

You need both, and they protect different parties. Advisory bounds protect the user, by producing a graceful result instead of a truncation. Enforced bounds protect the business, by making a runaway loop impossible rather than merely unlikely.

A design with only advisory bounds is a design that trusts a non-deterministic component to respect a budget — which Module 2 already established is not a control. A design with only enforced bounds works correctly and produces a worse experience every time a limit is reached, because the system had no opportunity to wrap up.

Bounds exist at layers you do not own

Worth knowing, because it affects diagnosis rather than design.

Your loop has a cap. The provider’s own tool-handling loop has its own iteration limit, which can pause a turn and require you to continue it. The model has a maximum output per response. A platform-level spend control may sit above everything. Four ceilings, and when a loop stops you need to know which one fired.

That is an observability requirement, not an architectural one, and it belongs in the plan: record which bound terminated each interaction. The aggregate is diagnostic gold — a rise in step-cap terminations means tasks are getting harder or the pattern is wandering, while a rise in timeout terminations means something upstream got slower.

Exhaustion is a designed path, not an error

The most common defect in this area is treating a hit limit as an exception.

When a bound fires, the system is in a defined state: some work done, some not, possibly some actions already taken. What happens next is a product decision — Module 3’s degradation ladder, with bound-exhaustion as one of the entry points. Hand off to a human with everything gathered so far. Report what was completed and what was not. Never leave a half-finished sequence of real actions with nobody informed.

And the cost of all of this, stated: bounds reduce capability. A loop with a tight step cap will fail at legitimately hard tasks that a looser one would have completed. The number is a trade, not a safety setting, and it belongs to whoever owns the outcome.

Deriving the numbers

The rubric asks for numeric bounds, which means they must come from somewhere.

• Step cap — observe the step count distribution on real tasks in a shadow run, then set the cap above the legitimate tail rather than at the median. Without data, start generous and tighten; starting tight hides what the real distribution is.
• Token budget — derive from the per-interaction cost ceiling. Cost ÷ the blended token rate gives you a token figure, and that is your budget.
• Timeout — take it from the latency requirement, not from the loop’s needs.
• Monthly cap — the approved budget, with a degradation threshold below it so you act before you hit it.

Four numbers, each derived

The constraints in play: $2.50 per interaction at p95, a 20-second action budget with visible progress, and an $8,000 monthly inference ceiling with degradation at 80%.

Bound	Value	Where it came from
Step cap	12 tool calls per interaction	Shadow-run observation: legitimate reschedules resolve in 3–6 steps; 12 is above the tail, not at the median
Task budget	Derived from the $2.50 p95 ceiling, expressed in tokens and given to the loop so it paces itself	Module 2’s cost requirement, converted at the measured blended rate
Wall-clock timeout	25 s, with progress visible from 2 s	Module 2’s action latency budget, plus a margin over the 20 s target
Spend cap	$8,000/month enforced outside the application, degrading to a cheaper tier at $6,400	Dev Raghunathan’s approved ceiling; the degradation point is 80%

On exhaustion: the interaction hands off to one of the 45 agents with the plan, the steps completed, the actions actually taken, and the reason the loop stopped. For an action already committed, the handoff says so explicitly — a half-done sequence that nobody is told about is how a support organisation loses trust in a tool permanently.

What gets recorded: which of the four bounds fired. Grace Adeyemi’s team will see these handoffs, and “the system gave up” is a very different conversation from “the system reached its cost limit on an unusually complex request”.

Note that only one of the four numbers is a safety limit. The other three are trades — capability, latency and experience given up in exchange for predictability — and each has an owner who should have agreed to it.

Feeds the workbook

The rubric requires loop termination and cost bounds specified numerically. Four numbers, each with its derivation, plus the exhaustion path. A bound with no stated origin is a guess with a decimal point.

Check your understanding

1. You have no shadow-run data yet. What do you set the step cap to, and what do you commit to doing once data exists?
2. Your loop has an enforced token cap but no advisory budget. Describe what the user sees when it fires.
3. Which of the four bounds would you expect to fire most often in the case system, and what would a rise in that rate tell you?

---

→ Bounds constrain how long the loop runs. The next question is what it can reach while running. — The tool surface