The Frontier Default, and Cascading

Overview

The matrix is built and one candidate leads on quality. The instinct is to take it and move on.

That instinct is often wrong, but not for the reason usually given — and the popular alternative, routing cheap requests to a smaller model, carries a cost almost nobody mentions. This topic puts the levers in the order measurement actually supports.

What the frontier default costs

Defaulting to the strongest available model means paying its rates on every request, including the trivial ones, and accepting its latency on every request, including the ones a user is watching.

In a system where most traffic is routine — and in a documented-question workload, most traffic is routine by construction — that is capability you are buying and not using. The waste is not dramatic per request. It is dramatic per month.

But the correction is not “use a cheaper model”. It is a sequence.

The lever order

graph TD
  A["<b>1 · Free wins</b><br/>caching, input hygiene,<br/>output length control<br/><i>no quality cost</i>"] --> B["<b>2 · Lower effort</b><br/>same capable model,<br/>reduced reasoning depth<br/><i>measure before assuming a cost</i>"]
  B --> C["<b>3 · Narrow-task models</b><br/>small models for classification,<br/>routing, extraction<br/><i>genuinely good fits</i>"]
  C --> D["<b>4 · Cascade</b><br/>cheap model first,<br/>escalate on low confidence<br/><i>structural cost — see below</i>"]

  classDef free fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef trade fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef struct fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class A free
  class B,C trade
  class D struct

Four levers in the order to try them. The terracotta one is where most teams start, and it is the only one that changes the architecture.

Lever 1 — the free wins. Caching, input hygiene and output-length control cost nothing in quality. Caching deserves particular attention because the economics are lopsided: reads from cache run roughly an order of magnitude cheaper than fresh input tokens, while writing to cache costs slightly more than sending the same tokens fresh. In a workload with a stable system prompt and a recurring corpus, that asymmetry is the single largest saving available, and it buys no accuracy loss whatsoever.

Caching is a prefix match, which has a design consequence: stable content must come first and volatile content last. A timestamp or a per-request identifier near the front of the prompt silently invalidates everything after it, and the symptom is a cache-hit rate of zero with no error anywhere. Verify hits by reading the cache-read token counts the provider returns, not by assuming.

Lever 2 — lower effort on the same model. Current models expose controls over how much reasoning they spend. Reducing that on a capable model frequently matches or exceeds what an older or smaller model achieves at full effort — which inverts the folk wisdom. Try this before any model swap, measure it on your golden set, and tune it per route rather than globally.

Lever 3 — small models for narrow tasks. Classification, routing, extraction and reformatting are genuinely well served by small models. These are not degraded general assistants; they are the right tool for a bounded job, and this lever is underused rather than overused.

Lever 4 — the cascade. Cheap model first, escalate when a confidence signal says the cheap answer is not good enough. Module 3 introduced this as cost-aware routing and named two costs: a router with its own failure modes, and two evaluation baselines instead of one.

The cascade cost nobody mentions

There is a third, and it can erase the saving entirely: caches are scoped to a model.

Split your traffic across two models and each one maintains its own cache. The shared prefix — your system prompt, your retrieved corpus material, your examples — is now cached twice, and each model only benefits from the share of traffic it actually receives. A cascade that sends 80% of requests to the cheap tier leaves the expensive tier with a cache that is cold more often, paying full input rates on exactly the requests that were already the expensive ones.

The interaction is worse than additive, and it means the honest comparison is not “cheap model rate versus expensive model rate”. It is blended cost with split caches versus single-model cost with one warm cache. Sometimes the cascade still wins. Often, at modest volume, it does not — and the only way to know is to measure both.

A related trap: changing reasoning effort mid-conversation can itself invalidate the cached prefix on some models. Lever 2 and lever 1 interact, so measure them together rather than in isolation.

Judge cost per completed task, not per request

The framing that prevents most mistakes here.

A cheaper request that needs a second attempt, a clarifying exchange, or an escalation to a human is not cheaper. Module 2’s unnecessary-handoff ceiling exists for exactly this reason: a model that declines more often looks excellent on a per-request cost chart and terrible on a cost-per-resolved-ticket chart.

So the denominator in your arithmetic is resolved interactions, not API calls. This is the same discipline as the business-driver chain from Module 1 — the unit has to be the one the business pays in.

The order applied

The ceiling is $1.54 per deflected interaction at p50 and $2.50 at p95, with a monthly inference ceiling of $8,000.

Lever 1 first. A stable system prompt plus the retrieved documentation is a large, repeating prefix, and in a corpus where 38% of tickets are already documented, repeat questions are the norm. Prompt caching is therefore the first and largest lever, with no accuracy cost. The design requirement that falls out: nothing volatile — no timestamp, no conversation identifier — may sit near the front of the prompt.

Lever 2 second. Measure a capable model at reduced reasoning effort against the 300-question golden set before considering any smaller model. If quality holds, this is a cost reduction with no new components, no router, and no second eval baseline.

Lever 3 where it fits. Routing and intent classification are bounded tasks; a small model is appropriate there regardless of what answers the questions.

Lever 4 last, and provisionally. Module 3’s bet 3 proposed small-model-first with escalation. The cache-scoping cost now argues for testing it rather than assuming it: at roughly 400 tickets a day — about 0.007 requests a second from the Module 3 scale calibration — cache warmth is not guaranteed on either tier, and splitting traffic could leave both caches cold. The falsification condition for bet 3 should be extended: if blended cost with split caches exceeds single-model cost with one warm cache, the cascade is abandoned.

That is a finding the Module 3 roadmap did not have, and it is exactly the kind of thing that justifies writing bets with reversal conditions rather than plans.

Feeds the workbook

Your AI stack selection should state the lever order you intend to apply, with the cascade’s three costs named if you adopt it. The rubric wants decisions that would be expensive to reverse identified — a cascade is one, because it doubles the eval surface.

Check your understanding

1. Your cache-hit rate is zero across repeated similar questions. Name three things that could be causing it, in the order you would check them.
2. Argue against the cascade for the case system using only cost reasoning. Now argue for it. Which argument would measurement settle?
3. A request costs half as much but needs a follow-up exchange 30% of the time. Is it cheaper? Show the arithmetic you would use.

---

→ Whichever model you chose, you are now married to a provider. What exactly have you married? — Vendor lock-in and the abstraction tax