Cost Governance

Overview

Four topics of arithmetic produce a number. This topic is about keeping it true.

A cost model is accurate on the day it is written and decays from then on. Usage patterns shift, a prompt grows, a cache breaks, traffic mix changes. Governance is the machinery that notices — and the reason it belongs in an architecture guide is that most of it is structural rather than procedural.

Attribution: you cannot govern a blended number

The foundation, and the thing most often missing.

A single API key shared across an organisation produces one bill covering several workloads. That number is ungovernable: you cannot tell which feature grew, you cannot compute cost per task for any one of them, and you cannot confirm whether a change you shipped actually saved anything.

So attribution is an architectural requirement, not a reporting nicety:

Per traffic class. The interactive answer path, the action path, and the batched evaluation and refresh work are three different workloads. Blending them produces an average describing none of them.
Per tenant, where the economics are per-customer. A unit cost that is fine on average can hide a handful of customers who are unprofitable.
Per tier, if Module 4’s cascade is in play. This is the only way to see a cheap tier quietly degrading while the blended figure stays green.
Per interaction, recorded in the trace Module 5 specified, which is what makes the other three computable at all.

The mechanism is separate keys or workspaces per workload plus per-request usage logging. Both are cheap to do at the start and awkward to retrofit, which puts them in the same category as the observability decisions from Module 5.

Three controls, doing three different jobs

graph LR
  A["<b>Visibility</b><br/>per-class, per-tenant<br/>attribution<br/><i>tells you</i>"] --> B["<b>Degradation</b><br/>cheaper tier at<br/>a threshold<br/><i>slows the bleed</i>"]
  B --> C["<b>Hard stop</b><br/>enforced spend cap<br/>outside the app<br/><i>ends it</i>"]

  classDef see fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef slow fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef stop fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class A see
  class B slow
  class C stop

Three controls in escalating order. Most systems have the first and the third and nothing in between — which means the only options are watching and switching off.

Visibility is attribution plus a threshold that notifies someone. It changes nothing on its own — and a cost alert nobody acts on is the same instrumentation theatre Module 3 identified with health checks.

Degradation is the middle control and the one usually missing. At a defined threshold the system moves to a cheaper configuration — a smaller model tier, a tighter retrieval depth, a reduced effort setting — trading quality for continuity deliberately. Module 2 required this as behaviour-on-approach rather than a bare ceiling, and the reason is that the alternative to graceful degradation is an outage.

A hard stop is an enforced cap that the application cannot exceed, ideally imposed by the platform rather than by the code it is protecting. The distinction matters: the failure mode it guards against is the application behaving unexpectedly, and a limit enforced inside the thing that might misbehave is not a limit.

Two properties make a hard stop usable rather than terrifying: it sits above the degradation threshold, so you degrade before you stop; and what happens when it fires is designed — which rung of Module 3’s ladder, and what the user sees.

Who owns the number

A governance question, and an architect’s answer is usually wrong if it is “me”.

Cost per interaction is an engineering-controlled quantity: prompt shape, cache design, step caps, retrieval depth. Cost per resolved ticket is a business quantity, because the denominator depends on the accuracy floor, the confirmation policy and the handoff rate — all of which belong to other people.

So the honest split is: engineering owns the numerator and reports it; whoever owns the business driver owns the unit; and the trade between them is a decision made jointly and recorded. Module 2’s NFR budget already named owners per line, and this is the same discipline applied after launch rather than before it.

The failure to avoid: cost as an engineering secret. A team that tracks spend privately and surfaces it only when something goes wrong trains the organisation to treat AI cost as an unpredictable hazard. Publishing the number routinely — per class, per month, against the ceiling — makes it an ordinary operational metric, which is the condition for being trusted with a larger budget later.

What governance costs

Attribution needs separate keys or workspaces and per-request usage logging. Degradation needs a second configuration that is tested, which means it is code nobody sees working. A hard stop needs somewhere outside the application to enforce it. And the reporting needs a person.

None of that is free, and on a small bill the honest finding may be that the full apparatus is not worth building yet. The minimum that is almost always worth it: per-class attribution and a hard stop. Those two mean you can find a problem and cannot be ruined by one.

Governance sized for the real numbers

The ceiling is $8,000 a month for inference, inside $1.2M a year approved overall. Dev Raghunathan’s stated fear is open-ended spend rather than the amount.

Attribution: three traffic classes with separate keys — the answer path, the action path, and batched eval/refresh work. Per-interaction cost recorded in the Module 5 trace, which makes per-class and per-tenant rollups computable without new plumbing. Per-tier attribution becomes mandatory the moment a cheap-model cascade ships.

Degradation at 80% — $6,400. At that point the answer path moves to a cheaper configuration and retrieval depth tightens. This was already in the Module 2 budget; what this topic adds is that the degraded configuration is a second code path which must be exercised, or it will be broken when it is needed.

Hard stop at $8,000, enforced outside the application. What happens when it fires is the degradation ladder’s lower rungs: documentation links without generation, then handoff to the 45 agents. Not an error page.

Ownership. Engineering owns and publishes cost per interaction monthly, per class. Dev owns the ceiling and the degradation threshold. The unit — cost per resolved ticket — is owned jointly with Maya Ortiz, because it moves whenever the accuracy floor or the handoff rate moves, and neither of those is an engineering decision.

The reporting habit that buys the most: a single monthly line showing cost per resolved ticket against the $3.00 driver, published whether or not anyone asked. Module 2’s arithmetic says that number will be uncomfortable for the foreseeable future — residual human cost alone exceeds the blended target — and publishing it routinely is what keeps that a known constraint rather than a discovery.

What is deliberately deferred: per-tenant attribution. At ~1,800 customers with a flat subscription, per-customer unit economics is not yet a question anyone is asking. The trigger to build it is the paid Copilot tier shipping, since a usage-priced tier makes per-tenant cost a product concern overnight.

Feeds the workbook

Your unit-economics model should end with the governance that keeps it true: what is attributed, where degradation triggers, where the hard stop sits, who owns which number, and what gets published. A model with no governance section describes a single day.

→ That closes the cost pillar. The second pillar fails even more quietly, because a successful attack returns a 200. — Threat modelling an LLM system