Measure Before Optimising

Overview

Module 6 built the measurement. This module spends it — and the first topic is about not spending it stupidly.

Premature optimisation is an old warning. It has a sharper edge here, for a reason specific to systems whose behaviour you do not fully control.

Why guessing is worse here than usual

In a deterministic system you can reason about where time and money go. The code path is fixed, so reading it tells you something true. An experienced engineer can often point at the hot spot before measuring and be right.

That inference does not survive non-determinism. The system chooses its own path: how many steps it takes, which tools it calls, how much it retrieves, how long its answer runs. Those choices vary per request and shift when a prompt, a model or a corpus changes. Reading the code no longer tells you where the cost is, because the code does not decide.

Three specific ways intuition misleads:

The median is not the bill. Module 6 established that a small share of interactions can carry a large share of spend. Optimising the typical case can leave the total almost unchanged.

The slow part is often not the model. Retrieval, guardrail calls, a tool waiting on the monolith, a confirmation waiting on a human — any of these can dominate. A latency problem assumed to be generation and actually caused by a tool call wastes a sprint.

Your fix may move the cost rather than remove it. Module 6’s clearest example: pruning the transcript to save context breaks the cache and costs more than it saves. Several optimisations in the next topic have this shape.

The discipline

graph LR
  A["<b>Profile</b><br/>per-step cost,<br/>latency, token split"] --> B["<b>Rank</b><br/>by ceiling, deflated<br/>for overlap"]
  B --> C["<b>One change</b><br/>free wins before<br/>trade-offs"]
  C --> D["<b>Measure</b><br/>cost and quality<br/>together"]
  D -->|"keep or revert"| B

  classDef first fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef mid fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef last fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  class A first
  class B,C mid
  class D last

Four steps and a loop. The loop returns to ranking rather than to profiling, because one change rarely invalidates the whole profile — and the keep-or-revert decision needs quality in the same view as cost.

Profile. Per-step cost and latency, the four-way token split from Module 6, and attribution per traffic class. Module 5’s trace already carries most of this, which is the payoff for having specified it.

Rank by ceiling. Size each candidate lever against the measured profile, and deflate overlapping ones so your shortlist cannot sum past the bill.

One change at a time. Free wins before trade-offs, and one lever per change so effects attribute. Two simultaneous changes produce a result you cannot assign to either.

Measure cost and quality together. This is the step that gets dropped, and dropping it is how a “saving” ships. A lever that cuts spend 15% and costs three points of correctness is not an optimisation; it is a trade for somebody with a business stake to accept or refuse.

Two rules that save the most time

Optimise cost per completed task, not per request. Module 6’s unit. A cheaper call that needs a retry, a clarification or a handoff is not cheaper, and in a system with a handoff path this is not a theoretical concern — it is the most likely way an optimisation backfires.

Price the tail. Compare options on the hardest tenth of real traffic. On easy cases every option looks adequate and the cheapest looks best; the decisions that matter are made on the cases where they differ.

What the discipline costs

Honestly: it is slower than just fixing the thing. Profiling requires instrumentation that may not exist yet, ranking requires arithmetic, and measuring each change requires evaluation runs that cost money and wall-clock time.

So calibrate. A change that is obviously free and obviously safe — deleting prose from a system prompt that restates a tool schema — does not need a ceremony. The full loop is for changes that trade quality, cost real engineering time, or alter structure. That is the same triage rule as Module 3’s design process, applied to optimisation.

And one permission worth granting yourself: “nothing here is worth changing” is a valid outcome. On a modest bill, with a comfortable latency budget, the honest shortlist is sometimes empty, and saying so beats manufacturing a lever to look diligent.

What the profile would have to show first

The targets: ≤ $1.54 per deflected interaction at p50, ≤ $2.50 at p95, p95 time to first token ≤ 1.5 s, p95 answer completion ≤ 8 s, and a 25-second ceiling on the action path with progress visible from 2 s.

There is no profile yet, which is the first finding. The platform has not run on real traffic, so every figure in Module 6’s cost model is an estimate labelled as such. The honest first move is not an optimisation plan — it is the shadow run that produces one.

What the profile must break out, because each points at a different lever:

Component	Why it matters
Retrieval latency and token volume	Retrieved documentation dominates input; it is also the chunking decision from Module 4 showing up as a bill
Guardrail calls	Module 6 noted these are billed calls on top of the call they protect — the line most cost models omit
Generation, split by the four token types	The actual model spend, and the cache hit rate
Monolith tool calls on the action path	A latency source that is not the model at all
Human confirmation wait	Minutes, not milliseconds — and the reason the action path wants the longer cache duration

The prediction worth recording before measuring, so it can be checked: on the action path, the dominant latency component is the human confirmation, and the dominant cost component is loop resending repriced by caching. If the profile contradicts that, the Module 6 arithmetic has a hole in it and finding out is worth more than any lever.

What is already known without a profile: the step cap of 12 bounds input volume to 78 turn-equivalents, and caching reprices most of that. Those are arithmetic rather than measurement, so they can be planned against now — which is why Module 6 put them in the model and not in this module’s plan.

Feeds the workbook

Your optimisation plan should open with what you would profile and what you predict, not with a list of techniques. The rubric requires every optimisation to state its quality cost — a plan written before a profile exists must say so, and say what would change it.

Check your understanding

1. For the case system, which component would you bet dominates p95 latency? Write the prediction down, then say what measurement would falsify it.
2. You have no telemetry and a deadline. Which single free win would you apply without measuring, and why is it safe?
3. An optimisation cuts cost 20% and correctness by two points. Whose decision is it, and what do you bring them?

---

→ The profile tells you where to push. Here is the catalog, and what each entry spends. — Optimisation techniques