Optimisation Techniques

Overview

The profile says where to push. This topic is what you can push with.

Module 6 covered the cost catalog in its free-wins-then-trade-offs order. This one adds the latency techniques, and asks a question about each that is easy to skip: is this actually your lever, or your provider’s?

Whose lever is it?

A useful first cut, because it saves effort on things you cannot change.

Speculative decoding — where a small model drafts tokens and a larger one verifies them, accepting the draft when it agrees — is a genuine latency technique with an elegant property: because the large model verifies every token, output quality is essentially unchanged. It is also, for hosted inference, somebody else’s implementation detail. Your provider may or may not use it; you generally cannot switch it on, tune it, or reason about it. It belongs in your vocabulary so you recognise it in a vendor conversation, and not in your optimisation plan.

That distinction generalises. Serving-level techniques — batching inside the server, kernel-level work, how requests are scheduled on a device — are levers if you self-host and vocabulary if you do not. Module 4 decided not to self-host, which removes a whole column of this catalog and is worth noting explicitly so nobody spends a week on it.

What remains genuinely yours: what you send, how much, to which model, at what reasoning depth, how often you can avoid sending it at all, and how much you ask back.

The latency catalog

graph LR
  A["<b>Yours</b><br/>prompt shape · caching ·<br/>retrieval depth · effort ·<br/>output length · parallelism"] --> C["Latency and<br/>cost outcome"]
  B["<b>Provider's</b><br/>speculative decoding ·<br/>serving batching ·<br/>device scheduling"] --> C

  classDef mine fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef theirs fill:#D8D3CB,stroke:#B8B2A8,stroke-width:2px,color:#3F3F3F
  classDef out fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  class A mine
  class B theirs
  class C out

Both branches affect the outcome; only one is a lever. The drained branch is vocabulary — know the words, do not plan around them unless you self-host.

Technique	What it buys	What it spends
Prompt caching	Large cost reduction; also latency, since cached prefix processing is faster	Prefix rigidity — volatile content must move after the breakpoint
Reduced reasoning effort	Lower latency and cost on the same model	Quality on hard cases; flat on easy workloads, steep on hard ones
Trimming retrieval depth	Direct input-cost and latency reduction	Recall — the answer may no longer be in what you sent
Output-length control	Output is the dearest token type, so this is both levers at once	Completeness, if specified carelessly
Request parallelisation	Wall-clock latency on independent work	Safety, where the work is not independent — see below
Distillation	A small fast model for a narrow task	The tail, and a snapshot you must maintain
Streaming	Transforms perceived latency without changing completion time	Incompatible with output checks that need the whole response

Two entries deserve more than a row.

Parallelisation, and where it is not safe

Running independent work concurrently is the cheapest latency win available when it applies. The discipline is in the word independent.

Safe: several retrievals against different sources; several read-only lookups; evaluating several candidate answers. Module 5’s tool surface made this concrete — read-only tools can be marked parallel-safe precisely so the harness may schedule them together.

Not safe: anything where one step’s result should inform the next, because parallelising it means the later step runs on stale information. And emphatically not safe for actions with effects, where concurrent execution can interleave into states neither call anticipated. Module 5’s advice stands: consequential tools are serialised, and the only reason the harness can tell which is which is that you promoted them to distinct tools.

One further trap: parallel calls each carry their own context, so parallelism trades token spend for wall-clock. It is a latency lever that can raise cost.

Distillation, and what it really is

Training a smaller model to imitate a larger one’s behaviour on a narrow task. It genuinely buys latency and unit cost, and for a bounded job it can be the right answer.

It is also fine-tuning wearing a different hat, which means Module 4’s bill applies in full: a dataset to build, an evaluation surface that grows, the least portable artefact in your stack, and a snapshot pinned to a base version while the frontier moves. Module 4’s reconsideration trigger — a narrow, stable, high-volume task with a measured prompting plateau — is the same gate here.

The honest framing: distillation is an optimisation that incurs a maintenance obligation. Most latency problems have a cheaper answer earlier in the table.

The ordering, restated

Free wins before trade-offs, and within trade-offs: effort before model, model before architecture. The reason to re-state it here is that latency pressure creates urgency, and urgency is what makes teams reach for the structural changes first.

If a latency budget is being missed, the order to try is: shape the output, check the cache is working, reduce retrieval depth, lower reasoning effort, parallelise what is genuinely independent — and only then consider a different model or a distilled one.

The plan, with one column empty

Against p95 time to first token ≤ 1.5 s and answer completion ≤ 8 s, with the cost ceilings from Module 6.

Lever	Expected effect	Quality cost	Status
Prompt caching, long duration on the action path	Largest single cost effect; some latency	None	Apply now
Output shaping — support answers are short	Cost and completion time	Risk of clipped answers if over-tightened	Apply now
Streaming end to end	Time to first token, perceived	Conflicts with whole-response output checks	Already decided in Module 2; the conflict is live
Parallel retrieval across docs and ticket corpora	Completion time	None, if genuinely independent	Apply — both are read-only
Reduced reasoning effort	Cost and latency	Unknown without the golden set	Blocked on evaluation
Cheap-tier routing	Cost	Accuracy on routed cases, plus split caches	Blocked on evaluation
Distillation	Latency on a narrow task	Dataset, maintenance, portability	Rejected for now; trigger is Module 4’s classifier case
Speculative decoding, serving batching	—	—	Not our lever — hosted inference

Two things to notice. The last row is there to be struck out: naming it prevents someone proposing it in month three, and Module 4’s decision not to self-host is what makes it unavailable.

And the streaming row is the one with a real unresolved conflict. Module 2 identified it, Module 6 priced the guardrail calls, and this plan has to pick: incremental checking on partial output, a cheap first-pass check before streaming begins, or a retraction path. Each costs something different, and the decision belongs with the compliance owner and the support lead jointly — not with whoever implements the guardrail.

Feeds the workbook

The rubric requires every optimisation to state its quality cost, not only its savings. Use the table shape above, and keep the “not our lever” rows in — a plan that quietly omits them invites the same proposal later.

Check your understanding

1. Which rows of your own plan are blocked on an evaluation existing? What does that imply about sequencing?
2. You parallelise two retrievals and cost rises. Explain why, and say whether it was still worth it.
3. A latency budget is being missed and someone proposes a distilled model. What do you try first, and what do you say about the proposal?

---

→ Optimisation assumes things work. The next three topics are about what happens when they do not. — Retry semantics