The Unit That Matters
Overview
Section titled “Overview”The previous topic gives you the cost of a request. Nobody buys requests.
This topic is about the unit a business actually pays in, the denominator you divide by, and the distribution you price against — three choices that decide whether your model survives contact with the person who approved the money.
Cost per completed task
Section titled “Cost per completed task”The unit is not cost per token, and it is not cost per request. It is cost per completed unit of business work — per resolved ticket, per handled conversation, per closed case.
The distinction has teeth because the cheapest request is frequently not the cheapest outcome:
- A cheaper model that fails bills its tokens, then bills the retry, then costs whatever the failure costs downstream.
- A response that triggers a clarifying exchange costs two interactions, not one.
- An answer that ends in a handoff costs the interaction plus the human handling cost that deflection was supposed to avoid.
So a change that lowers cost per request while raising the handoff rate can raise cost per resolution. Module 2 anticipated this with the unnecessary-handoff ceiling; here it becomes arithmetic. A model that optimises the numerator while ignoring the denominator will recommend the wrong design.
Choosing the denominator honestly
Section titled “Choosing the denominator honestly”The denominator is where cost models get flattering, usually without anyone intending it.
The temptation is to divide by attempted interactions, which makes the number look good: every cheap attempt that failed is still in the denominator, diluting the expensive ones. The honest denominator is work actually completed to the standard the business asked for.
Three tests for a denominator:
- Would the business recognise it? If the unit is “model calls”, nobody outside engineering can use the number.
- Does it include the failures? Attempts that ended in a handoff consumed cost and produced no deflection. They belong in the numerator and not in the denominator.
- Is it one traffic class? Blending an interactive path with a nightly batch job produces an average describing neither. Cost per task means nothing across classes — model them separately.
Price the tail, not the median
Section titled “Price the tail, not the median”The most consequential modelling error in this area, and the one that produces the biggest surprises.
On a typical interaction every design looks similar. The bill is decided by the hard tail — and the concentration is severe. In one published analysis of a twenty-problem workload, two problems carried 43% of the total spend. That is not an outlier; it is the normal shape of a workload where the model decides how much work to do.
Two practical consequences.
Model p50 and p95 separately, and budget against p95. Module 2 required both numbers for exactly this reason. A design whose median cost is comfortable and whose p95 is fifty times higher has a tail problem, and the tail is where the monthly ceiling gets breached.
Compare options on the hardest tenth of the workload. On easy cases everything looks adequate and the cheapest option looks best. The differences that matter show up on the cases the cheap option cannot handle — which is also where Module 4’s model-selection evidence should have been gathered.
Say what is measured and what is estimated
Section titled “Say what is measured and what is estimated”A discipline that costs nothing and earns disproportionate credibility: an estimate and a measurement must not look the same in your document.
graph TD A["Billing or usage data<br/>for this workload"] --> M["<b>Measured</b><br/>quote in dollars"] B["Application's own<br/>usage logs"] --> P["<b>Partly measured</b><br/>quote as % of bill,<br/>dollars in parentheses"] C["Reading the code,<br/>no telemetry"] --> E["<b>Estimated</b><br/>quote as relative bands:<br/>large / medium / small"] classDef src fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff classDef best fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff classDef mid fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff classDef weak fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff class A,B,C src class M best class P mid class E weak
Three evidence tiers, three ways of quoting a number. Quoting a code-read estimate to the cent is how a cost model loses its authority permanently.
The related discipline: overlapping savings cannot be added. Two levers that act on the same tokens — caching a large inlined document, or removing the document — claim the same money. Size each one independently, then deflate the lower-ranked ones for their overlap with the higher, so your total can never exceed the bill. A cost model whose savings sum to 120% of spend is one somebody will stop reading.
And a finding worth permitting yourself: “nothing here is worth changing” is a valid result. On a small bill, the honest shortlist is sometimes empty, and saying so is more useful than manufacturing a lever.
Feeds the workbook
Section titled “Feeds the workbook”The rubric requires arithmetic with stated assumptions rather than an estimate, and that the model says plainly whether the cost-per-ticket target in your brief is achievable — including if the answer is no. Name your denominator explicitly, model p50 and p95 separately, and label every figure by evidence tier.
→ The tail is where the money is. In a loop, the tail has a specific and steep shape. — Why loops cost what they do