Cost as a Requirement

Overview

Accuracy and speed both have numbers now, and both cost money. In a system with a model in it, the money is metered by the word — which makes cost a requirement that shapes structure rather than a line item reviewed after launch.

Module 1 made the claim that cost becomes a structural design parameter. This topic turns it into a requirement with arithmetic behind it.

Why this is genuinely new

A conventional service costs roughly the same per request, and most of that cost is capacity you provisioned in advance. You size it, you pay for it, and individual requests are close to free at the margin.

A token-metered call inverts both properties. Cost accrues per request, and it scales with decisions the architecture makes: how much retrieved context goes in, which model handles it, how long the answer is, how many times an agent loops, how many checks run alongside. The marginal request is not free, and its price is a function of your design.

Two consequences that have no equivalent in the deterministic world:

Cost per request is unbounded unless you bound it. An agent that can call tools and re-plan has no natural stopping point. One pathological conversation can cost a hundred times the median.

Users influence your unit cost. Longer questions, longer histories and more documents retrieved all cost more. Nobody can make your database charge you more by typing a longer sentence; here they can.

graph TD
  C1["Retrieved context<br/>size"] --> CPI["<b>Cost per<br/>interaction</b>"]
  C2["Agent steps<br/>and retries"] --> CPI
  C3["Model tier<br/>chosen"] --> CPI
  C4["Output length"] --> CPI
  C5["Guardrail and<br/>judge calls"] --> CPI
  CPI --> M["<b>Blended cost<br/>per handled ticket</b><br/>the business unit"]
  M --> MC["Monthly ceiling<br/>the approved budget"]

  classDef yours fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef given fill:#7C7C7C,stroke:#5E5E5E,stroke-width:2px,color:#ffffff
  classDef unit fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class C1,C2,C3 yours
  class C4,C5 given
  class CPI,M,MC unit

Five inputs to one unit cost. The amber three are architectural decisions, which is what makes a cost ceiling a structural requirement rather than a budget line.

Derive the ceiling, do not estimate it

The arithmetic runs in the direction the chain from Module 1 demands: from the driver down to the unit cost, never from a vendor price list up.

The method, in four steps:

1. Take the business driver’s unit target — the cost per unit of work the business committed to.
2. Establish what share of the work the system can actually address, and be ruthless here; the rest keeps costing what it costs today.
3. Subtract the cost of the work that still reaches a human, including the share the system attempts and hands off.
4. What remains, divided by the volume the system handles, is the allowed cost per interaction. That is your requirement.

Run honestly, this arithmetic often tells you the target cannot be met the way everyone assumed — and finding that out in week five is the single most valuable thing an architect can do with an afternoon.

Three numbers, not one

Cost per interaction at p50 and p95. Both, because the tail is the risk. A median of a few cents with a p95 fifty times higher means a handful of conversations can eat the budget, and bounding that tail is a structural requirement: a step limit, a token cap, a retry ceiling.

Blended cost per unit of business work. The number the business driver is written in. This is the one that goes in front of whoever approved the money.

A monthly ceiling with defined behaviour on approach. Not “we expect to spend about this”. A ceiling is only a requirement if the system does something when it nears it — degrade to a cheaper model, narrow the eligible request types, queue, or stop. Whoever holds the budget fears open-ended spend far more than they fear a conservative cap, and a requirement that names the degradation buys more trust than any forecast.

Cost levers are structure, not configuration

Each lever below changes what the system is, and each one spends a different quality attribute. This is a preview — Module 6 builds the FinOps model properly — but the requirements have to be written now, because Module 3 picks a structure.

Lever	What it buys	What it spends
Cheap model first, escalate on low confidence	Large unit-cost reduction	Accuracy on the cases the small model gets wrong, plus a confidence signal you must trust
Caching, exact or semantic	Repeat questions become nearly free	Staleness, and a hard question about what counts as the same question
Trimming retrieved context	Cost scales with context size, so this is direct	Recall — the answer may no longer be in what you sent
Bounding agent steps	Caps the tail	Some genuinely hard cases now fail and hand off
Batching	Better throughput economics	Latency, which you already budgeted

Every row spends a requirement from an earlier topic. That is not a flaw in the table; it is why the next topic exists.

Deriving the ceiling, and finding the target cannot hold

The driver: cost per ticket from $6.40 toward $3.00. Volume ~12,000 tickets a month. The audit says 38% — about 4,560 — are answerable from existing documentation.

First, test the target as stated. Suppose the platform handled every one of those 4,560 for free. The remaining 7,440 still cost $6.40 each, which is $47,616, against a blended target of 12,000 × $3.00 = $36,000.

$47,616 > $36,000. The company-wide $3.00 target is unreachable by deflecting the 38%, even at zero AI cost. Blended cost lands at $3.97 in the perfect case, and more realistically $5.11.

That is a finding, not a failure. It says the requirement has been written against the wrong denominator, and it gives you three honest options to put in front of Dev Raghunathan: restate the target as applying to the deflectable segment, extend deflection beyond the documented 38%, or reduce the cost of tickets that still reach a human with agent-assist. Only the third one moves the company-wide number much.

Second, derive the unit ceiling for the segment. Take $3.00 as the target for the 4,560 answerable tickets, and assume 70% are handled without a human while 30% hand off at $6.40:

0.70 × X + 0.30 × $6.40 = $3.00
0.70 × X = $1.08
X = $1.54 per deflected interaction

So the requirement reads: ≤ $1.54 cost per deflected interaction at p50, ≤ $2.50 at p95, with a monthly platform ceiling of $8,000 and automatic degradation to a cheaper model tier at 80% of it.

Worth noticing that $1.54 is generous — comfortably above a typical retrieval-and-answer interaction. The binding constraints here are not the median but the tail and the ceiling, which is exactly why the requirement needs all three numbers. The $1.2M a year Dev approved — about $100,000 a month — covers build and run, not just inference; the $8,000 is the inference slice.

What this costs you

Deriving a ceiling publicly means publishing the assumptions it rests on, and some of them will be wrong. Expect to be corrected on the deflection rate in particular.

Publish them anyway. The alternative is a cost requirement nobody derived, which will be discovered to be unreachable at the point where the structure can no longer change — which is the expensive version of the same conversation.

Feeds the workbook

The rubric asks that your cost NFR be derived arithmetically from the cost-per-ticket driver in your brief. Show the working, including the step where the stated target does not survive contact with the arithmetic. A cost ceiling with no derivation is a number you chose.

Check your understanding

1. Redo the segment derivation at a 50% handling rate instead of 70%. What happens to the allowed cost per interaction, and what does that tell you about which assumption matters most?
2. Your p50 cost is comfortable and your p95 is forty times higher. Which lever from the table do you reach for, and which requirement does it spend?
3. The monthly ceiling is reached on the 22nd. Write the requirement describing what the system does for the rest of the month.

---

→ Cost, accuracy and speed are specified. Now: what the system must never be talked into doing. — Safety and alignment requirements