Inference Economics

Overview

Whichever way the previous topic went, the cost structure does not resemble an ordinary service. Hosted inference is metered per token; self-hosted inference is paid for by the hour whether or not anyone uses it.

Both have an arithmetic you can do in an afternoon, and doing it is what separates a defensible unit-cost claim from a guess.

Hosted: cost is a function of prompt shape

Hosted — the bill is built from the request itself:

graph LR
  I["Input tokens<br/><i>retrieved context<br/>dominates</i>"] --> C["× input rate"]
  O["Output tokens<br/><i>priced at a multiple<br/>of input</i>"] --> D["× output rate"]
  C --> T["Cost per<br/>interaction"]
  D --> T

  classDef in fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef out fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef tot fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class I,C in
  class O,D out
  class T tot

Self-hosted — the bill is built from the clock, and usage only changes the denominator:

graph LR
  H["Hours provisioned<br/><i>730 per month,<br/>used or not</i>"] --> R["× hourly rate"]
  R --> U["÷ interactions<br/>actually served"]
  U --> T2["Cost per<br/>interaction"]

  classDef fixed fill:#7C7C7C,stroke:#5E5E5E,stroke-width:2px,color:#ffffff
  classDef div fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef tot fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class H,R fixed
  class U div
  class T2 tot

Two different arithmetics. In the first, reducing usage reduces the bill. In the second, it does not — it only raises the cost of each request you did serve.

Three consequences of the hosted shape, all architectural:

Output costs more than input. Output tokens are priced at a multiple of input tokens — commonly several times. That makes answer length a cost lever, which makes response-length control an architectural concern rather than a stylistic one.

Retrieved context dominates input volume. Which means chunking strategy and retrieval depth are cost decisions, not just quality decisions. Two topics from now this becomes concrete.

Caching asymmetry is the biggest free lever. Reads from cache run roughly an order of magnitude cheaper than fresh input; writes cost slightly more than fresh. In a workload with a stable prefix, that is the largest saving available at zero quality cost — and it is why prefix stability is a design rule, not an optimisation.

One more lever worth knowing: asynchronous batch processing commonly carries a standing discount, often around half price, in exchange for latency measured in minutes or hours rather than seconds. Useless for a user waiting on an answer; excellent for backfilling an index, generating evaluation runs, or any bulk work with no human attached.

Self-hosted: cost is utilisation

The arithmetic is one division, and it is unforgiving:

cost per interaction = (hourly rate × hours provisioned) ÷ interactions served

Hours provisioned is 730 per month if the service is always on, and it must be always on if users expect answers at any hour. Interactions served is whatever your real volume turns out to be. The ratio between the compute you pay for and the compute you use is your utilisation, and at low volume it is brutal.

This is also where Module 3’s scale calibration earns its place: you cannot do this arithmetic without the real arrival rate, and the real arrival rate is usually far lower than the headline volume suggests.

Quantization, and what it actually costs

Quantization reduces the numerical precision of a model’s weights, cutting memory and raising speed — the standard lever for making self-hosting viable on cheaper hardware.

Its cost is quality, and the important detail is that the cost is not evenly distributed. Degradation concentrates in the hardest cases: long reasoning chains, unusual phrasings, edge cases, anything requiring precision. Average-case performance can look nearly unchanged while the cases you most needed handled get worse.

Which has a direct methodological consequence. Evaluate quantized variants against your golden set — the one that deliberately contains the hard cases — and look at the confident-error rate specifically, not the aggregate correctness. A quantized model that holds average accuracy while doubling confident errors has failed the Module 2 requirement that matters most, and an aggregate number will not show you that.

Doing the division, and finding the real blocker

Volume: ~12,000 tickets a month, 38% answerable from documentation — about 4,560 — of which 70% are handled without a human, so roughly 3,192 deflected interactions a month.

The self-hosted arithmetic. Stated assumption, not a case fact: an accelerator instance at $2.00 per hour — verify current rates before using this.

$2.00 × 730 hours        = $1,460 / month
$1,460 ÷ 3,192           = $0.46 per deflected interaction
With a second instance for availability: $0.91

Both land under the $1.54 p50 ceiling, and $2,920 a month sits well under the $8,000 inference ceiling. So compute cost does not decide this — which is not the answer most people expect.

Now the utilisation. At roughly 8 seconds of work per interaction, 3,192 interactions is about 7 hours of actual compute out of 730 provisioned — under 1% utilisation. You would be paying for idle accelerators almost all of the time, and that is before redundancy.

What the arithmetic actually reveals. The binding constraints are not cost at all. They are the operational capability nobody has, the capability gap between open-weight and frontier models against Maya Ortiz’s 90% correctness floor, and ownership of availability for a workload the support organisation depends on across three time zones.

This is the Module 3 scale-calibration lesson arriving a second time, and it is worth stating as a general caution: when the arithmetic comes out comfortable, that is not permission to proceed — it is a signal to check which constraint is actually binding. Here the honest finding is that the self-hosting question was never a cost question.

Feeds the workbook

Your AI stack selection should carry this arithmetic for whichever path you chose, with assumptions labelled. If you self-host, state the utilisation. If you host, state which levers get you under the ceiling and what each costs in quality.

Check your understanding

1. Redo the self-hosted division at ten times the volume. Does the conclusion change, and which constraint binds then?
2. Output tokens cost several times input tokens. Name two design changes that follow, and what each gives up.
3. A quantized model holds aggregate accuracy and doubles confident errors. Which Module 2 requirement did it break, and how would you have caught it?

---

→ Cost is settled. Capacity is a different question, and planning it here does not resemble planning an ordinary service. — Throughput and concurrency