Tactics for Probabilistic Parts
Overview
Section titled “Overview”The catalog in the previous topic was assembled for components that behave the same way twice. Every tactic in it assumes failure is an event: the thing is up or down, fast or slow.
A probabilistic component fails differently. It stays up, stays fast, and is wrong. Two tactics address that directly, and both are structural rather than configurable.
Tactic 1 — degradation to a deterministic fallback
Section titled “Tactic 1 — degradation to a deterministic fallback”Graceful degradation already exists in the catalog. The addition here is the floor: the bottom rung of the ladder must not contain a model.
graph TD A["Full generated answer<br/>grounded in retrieved docs"] --> B["Narrower answer<br/>fewer claims, cited only"] B --> C["Retrieval only<br/>the matching documents,<br/>no generation"] C --> D["Human handoff<br/>with gathered context"] D --> E["Static response<br/>contact details,<br/>known-good links"] classDef prob fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff classDef mid fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff classDef det fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff class A prob class B,C mid class D,E det
Five rungs, descending. The olive rungs work when every model in the system is unavailable or untrusted — which is the property that makes this a tactic rather than an aspiration.
Three design rules follow, and each one is the kind of thing that gets skipped:
The bottom rung has no model in it. If your last line of defence calls a model to decide what to say when models are failing, you have a single point of failure drawn as a fallback.
Each rung is independently reachable. The ladder must be enterable at any level — not only by descending from the top. A confidence signal, a provider timeout and a compliance block are three different reasons to arrive at rung three, and each should get there directly.
Degradation is visible. The system must record which rung served a request, because the aggregate is your early warning that something upstream is quietly failing. This is the countermeasure to the quiet-failure problem from Module 1: without it, a system that has silently dropped to rung three looks exactly like a healthy one.
What it costs: a second, third and fourth code path, none of which carries a visible feature, all of which must be tested. And a product decision about whether a degraded response counts as success — which is a conversation with whoever owns the business metric, not a technical choice.
Tactic 2 — cost-aware routing
Section titled “Tactic 2 — cost-aware routing”The move: send each request to the cheapest component that can handle it, and escalate only when a signal says the cheap path is not good enough. A small model answers most requests; a larger one handles the rest.
This is a cost tactic, and Module 2’s arithmetic is what justifies it. Where a hard per-interaction ceiling exists, routing is usually the single largest lever available — larger than prompt tuning, larger than context trimming.
Four structural consequences, which is what makes it architecture rather than configuration:
- A router exists. It is a new component with its own failure modes and its own latency contribution.
- A confidence or difficulty signal exists, and something must produce it. The weak link in every routing design is the quality of this signal, and its calibration is a measurement problem, not an assumption.
- The routing policy lives somewhere changeable. Thresholds that get tuned weekly cannot be constants in a prompt.
- Observability is per-tier. Aggregate accuracy across a cascade hides the case where the cheap tier has quietly degraded and the expensive tier is carrying it.
What it costs, plainly: two models means two evaluation baselines. The golden-set work from Module 2 roughly doubles, because “the system is 90% correct” is no longer a single claim. Escalation also adds latency for the requests that take both paths — you pay the cheap call and the expensive one, so a badly calibrated router is worse than no router on both axes at once.
A third, smaller addition: bounding
Section titled “A third, smaller addition: bounding”Deterministic components have natural stopping points. A component that can loop, re-plan and call tools does not, which makes bounds a structural element rather than a safety net: a step cap, a token cap, a wall-clock timeout, each with defined behaviour on exhaustion.
Module 2 put numbers on the p95 cost. This is where those numbers become components.
Fallbacks you never exercise do not work
Section titled “Fallbacks you never exercise do not work”The uncomfortable corollary. Every path in the ladder below rung one runs rarely, which means it is the least-tested code in the system and the code you depend on in the worst moment.
The response is to exercise it deliberately — scheduled degradation, fault injection on the provider call, a share of traffic deliberately routed to the lower rungs. The cost is engineering time spent on something that produces no feature, in a company where capacity is already contested. State it as a requirement anyway, because the alternative is discovering in an incident that rung three has been broken for five months.
Feeds the workbook
Section titled “Feeds the workbook”Both tactics belong in your style selection as named choices with costs, and the ladder shapes the reference architecture — rungs three to five are components, not error handlers. Module 7 returns to all of this as a full uncertainty policy.
→ You have styles, boundaries and tactics. Now make the actual choice — inside an estate you did not build and a capacity you do not control. — Choosing under constraint