Latency and Perceived Time

Overview

Correctness has a number now. Users meet something else first: the wait.

Latency requirements for generated responses do not work like latency requirements for an API, because the response arrives gradually and because the thing producing it is not yours. This topic is about which number to write down, and what writing it down forbids.

One requirement is three numbers

A conventional service has one meaningful timing figure: how long until the response is complete. A streamed response has at least three, and they have different owners and different fixes.

graph LR
  R["Request<br/>accepted"] --> T1["<b>Time to first token</b><br/>when the user stops<br/>wondering if it works"]
  T1 --> T2["Streaming rate<br/>how fast it reads"]
  T2 --> T3["<b>Completion</b><br/>when the answer<br/>is whole"]
  T3 --> T4["Time to resolution<br/>including handoff<br/>and follow-ups"]

  classDef plain fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef key fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef whole fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  class R,T2 plain
  class T1,T3 key
  class T4 whole

Four measurement points on one interaction. The amber pair is what the user feels; the olive one is what the business driver actually cares about.

Time to first token is the number users judge the system by. The gap between pressing send and seeing anything is the only part of the wait that feels like a wait; once text is moving, tolerance stretches remarkably.

Streaming rate matters only in that it must exceed reading speed. Beyond that, making it faster buys nothing a user notices.

Completion time matters to anything downstream that needs the whole response — including, critically, an output check.

Time to resolution is the number the business driver is actually about. A four-second answer that fails to resolve the question costs more than a twelve-second one that works, because the follow-up and the eventual handoff are where the money goes. If your latency requirement optimises the first number and ignores this one, you will hit the target and miss the point.

Percentiles, and the part you do not control

Use p95 or p99, never averages — Module 1’s reader knows why for ordinary services, and here there is an extra reason. Generation latency has a fat tail driven by response length and by the provider’s load, neither of which you control.

That last clause changes the shape of the requirement. A latency requirement against an upstream you do not own is incomplete unless it states what happens when the upstream misses it:

p95 time to first token ≤ 1.5 s. If no token has arrived within 10 s, abandon the attempt and hand off, with the gathered context attached.

Written this way, a performance requirement has quietly become an availability requirement, and it has forced three structural elements into existence: a timeout, a fallback path, and a context package that survives the abandonment. That is what it means for a quality attribute to shape structure — and it is why the environment part of the scenario is where the design work hides.

Long-running work needs a different contract

An assistant that answers is a request-response interaction. An agent that takes an action — checking a schedule, making a change, verifying it — may legitimately take much longer than any latency budget a user would accept in silence.

Stretching the latency requirement to cover it is the wrong move. Swap the contract instead: for operations beyond a few seconds, the requirement is progress, not speed. The system must say what it is doing, must be interruptible, and must leave the user able to walk away and come back. The mechanics are ordinary asynchronous design and are assumed knowledge here; what is new is remembering to write it as a requirement rather than discovering it in usability testing.

The conflict nobody mentions until implementation

Streaming and output checking pull against each other, and this is the clearest example in the module of a requirement pair that cannot be averaged.

Streaming means emitting text before the response exists. An output guardrail that must see the whole answer before releasing it means not emitting anything until it does. You can have a fast first token, or you can have a fully checked response, and the honest resolutions are all compromises: check incrementally on partial output, stream only after a cheap first-pass check, or accept a retraction path where text already shown can be withdrawn.

Each has a cost. Incremental checking catches less. Delayed streaming spends the latency budget you were protecting. Retraction is visible to the user and looks exactly like a system that does not know what it is doing.

Name this conflict in the requirements rather than leaving it for whoever implements the guardrail, because they will resolve it alone and in favour of whichever requirement their ticket mentions.

A latency budget that forces structure

Requirement	Value	Environment
p95 time to first token	≤ 1.5 s	Normal operation
p95 answer completion	≤ 8 s	Normal operation
Hard timeout to handoff	10 s with no first token	Primary provider degraded
Action confirmation with progress	≤ 20 s, with visible progress after 2 s	Rescheduling a job or resetting a device
p95 time to resolution	≤ 4 min, against a support baseline measured in hours	All

Two notes on where these came from. The handoff timeout is sized against the 45 agents across three time zones: a handoff at 02:00 in the covering region is not equivalent to one at midday, so the requirement is incomplete without Grace Adeyemi’s coverage schedule beside it. And the resolution target is the one tied to the driver — first-response time is one of the four business drivers, and it is the only row here that a CFO would recognise as connected to the $6.40 → $3.00 case.

The structural consequences, visible before any design exists: streaming end to end with no buffering intermediary, a timeout and fallback on the model call, a context package that survives abandonment, and a progress channel for actions. Four structural elements from one table.

Feeds the workbook

Your NFR budget needs a p95 latency budget for streamed responses. Write the timeout and the fallback in the same row — a latency number without them is not yet a requirement, because it does not say what the system does when it is missed.

Check your understanding

1. Your p95 time to first token is met and users still describe the assistant as slow. Which of the four numbers would you look at first, and what would you expect to find?
2. Write the streaming-versus-output-checking conflict as a register entry: the two requirements, who holds each, and who decides.
3. An action takes 45 seconds. Write the requirement that makes that acceptable without raising the latency budget.

---

→ Faster and more accurate both cost money, and here the money is metered by the word. — Cost as a requirement