Pricing Models

Overview

Module 4 established that cost is a function of prompt shape. This section builds the model you defend to whoever controls the budget, and it starts with the shapes the bill can take.

Four pricing models, each implying a different architecture. And inside the most common one, four separately-priced quantities that most cost models collapse into two.

Four shapes of bill

Per token. You pay for what each request contains and produces. Cost scales with usage, the marginal request is never free, and the architecture controls the bill through prompt shape. This is the default for hosted inference and the rest of this topic is mostly about it.

Per hour. You rent capacity. Cost is fixed, usage only changes the denominator, and the governing number is utilisation — Module 4 did this arithmetic and found under 1% utilisation at the case volume.

Provisioned throughput. You reserve a guaranteed rate and pay for the reservation. It buys predictability — a known bill and a known ceiling that no traffic spike can breach — and it costs you the flexibility of paying for what you use. Structurally it suits high, steady, predictable volume and punishes spiky workloads, because you are paying for the peak you reserved during every trough.

Batch. Work with no human waiting is commonly discounted substantially — around half price is typical — in exchange for asynchronous completion. The important detail is that the discount applies to every token in the request, including cached reads and writes, so it stacks with caching rather than competing with it.

One caution on batch: the completion window is an expiry, not a service level. “Within 24 hours” means it will not be later than that, not that it will be prompt. Anything a user is waiting for stays synchronous.

Four token types, not two

Most cost models have a line for input and a line for output. A request with caching has four separately-priced quantities, and conflating them produces a model that is wrong in both directions.

graph LR
  A["Regular input<br/><b>1×</b><br/><i>uncached, full price</i>"] --> T["Cost of one<br/>request"]
  B["Cache write<br/><b>~1.25× or ~2×</b><br/><i>higher for longer-lived<br/>entries</i>"] --> T
  C["Cache read<br/><b>~0.1×</b><br/><i>the whole point</i>"] --> T
  D["Output<br/><b>several× input</b><br/><i>length is a cost lever</i>"] --> T

  classDef full fill:#7C7C7C,stroke:#5E5E5E,stroke-width:2px,color:#ffffff
  classDef write fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef read fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef out fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class A full
  class B write
  class C read
  class D out
  classDef tot fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  class T tot

Four quantities, four rates, spanning roughly a fiftyfold range from cache read to output. Multipliers are structural and approximate — verify the current figures in the provider’s pricing documentation before putting them in a model.

Three consequences for how you model cost.

Output is the expensive line. Priced at a multiple of input, commonly several times. That makes response length an architectural concern — specify the output shape, ideally with an example, rather than hoping for brevity.

Cache reads are the cheap line, by an order of magnitude. Which is why Module 4 called prefix stability a design rule. A workload with a large stable prefix and a high hit rate is operating on a different cost curve from the same workload with a cold cache.

Cache writes cost more than sending the tokens fresh, and longer-lived cache entries cost more to write than short-lived ones. This matters because it means caching is not unconditionally free: a prefix written and never read again is a surcharge. The lever only pays when the entry is read, which makes hit rate the number to watch rather than the presence of caching in your code.

The decision the pricing model forces

One structural choice falls directly out of this topic, and it is worth making explicitly rather than by default.

A longer-lived cache entry costs roughly twice a short-lived one to write, and pays for itself the first time it prevents a miss — because a miss means resending the entire prefix at full price and writing it again. So the decision is governed by the gap between successive requests in a conversation.

Which has an unexpected consequence for anything with a human in the loop. A design that pauses to ask a person for confirmation — exactly what Module 5’s action boundary requires for consequential actions — has gaps measured in minutes, not milliseconds. A short-lived cache will have expired by the time the person answers, so every confirmation turns into a full-price cache miss and rewrite.

That makes cache duration a function of your oversight design, which is not a connection most cost models notice.

What the bill is made of

The platform’s inference spend, with the structure made explicit. Target: ≤ $1.54 per deflected interaction at p50, ≤ $2.50 at p95, with a monthly inference ceiling of $8,000.

Pricing model: per token, hosted, in-region. Module 4’s three gates settled this; per-hour was rejected on capability and operations rather than cost, and provisioned throughput makes no sense at roughly 400 tickets a day with pronounced time-zone troughs — you would be reserving a peak and paying for it overnight across three regions.

Where batch applies, and where it does not. Customer-facing answers are synchronous by definition. But three workloads here have no human waiting and should be batched at the discount: the weekly embedding refresh over the documentation corpus, the evaluation runs against the 300-question golden set, and any backfill over historical tickets. Those are not small — eval runs in particular are recurring — and the discount stacks with caching.

The four token types, for one deflected answer. A stable system prompt plus retrieved documentation forms the cached prefix; the customer’s question is uncached input; the answer is output at several times the input rate. The cost model needs all four lines, because the levers in two topics’ time act on them differently.

And the oversight interaction. Every credit requires a human confirmation from one of the 45 agents, across three time zones. A confirmation arriving minutes later — or, at 02:00 in a thinly covered region, considerably later — means a short-lived cache entry has already expired. So the action path wants the longer cache duration despite its higher write cost, while the answer path, where responses are immediate, does not.

Two different cache durations for two paths, decided by Grace Adeyemi’s coverage schedule rather than by anything in the pricing page. That is the kind of coupling a unit-economics model only surfaces if it is built from the four quantities rather than from an average.

Feeds the workbook

Your unit-economics model should have four input lines rather than two, and should state which workloads batch. The rubric asks for arithmetic with stated assumptions — the multipliers are assumptions, so date them and name their source.

Check your understanding

1. Which workloads in the case system have no human waiting? What would batching them save, and what would it complicate?
2. Your cache hit rate is 20%. Are you saving money or paying a surcharge? Show the reasoning.
3. Provisioned throughput is proposed to make the bill predictable. What would you ask about the traffic shape before agreeing?

---

→ Four token types give you a cost per request. The business does not buy requests. — The unit that matters