Throughput and Concurrency

Overview

Capacity planning for an ordinary service starts with requests per second. For inference that is the wrong unit, and using it produces plans that are wrong by orders of magnitude in both directions.

Three properties make this different. None is exotic, and all three are routinely missed.

Concurrency, not requests per second

An ordinary request occupies capacity for milliseconds. An inference request occupies it for seconds — and the whole time, it is holding something.

So the unit that matters is how many requests are in flight at once:

graph LR
  A["Arrival rate<br/><i>requests per second</i>"] --> C["× duration<br/><i>seconds per request</i>"]
  C --> N["= concurrent<br/>requests in flight"]
  N --> P["+ headroom<br/>for bursts"]
  P --> L["Check against the<br/>limit that binds:<br/>quota, or batch capacity"]

  classDef in fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef calc fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef out fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  class A,C in
  class N,P calc
  class L out

Arrival rate times duration gives concurrency. It is one multiplication, it takes a minute, and it routinely overturns a capacity plan.

At low arrival rates this number is startlingly small, and the planning instinct — provision for the headline volume — overshoots enormously. At high arrival rates with long responses, it grows faster than people expect, and the same instinct undershoots.

Duration is not known in advance

The second property. Response duration is dominated by output length, and output length is not known when the request arrives.

This gives inference workloads a duration variance unlike a service doing bounded work. A request might take two seconds or forty, and nothing upstream can predict which. Two consequences:

Queueing behaves badly. Long requests block, and tail latency degrades faster than arrival-rate growth alone would suggest. This is the mechanism behind the fat tail Module 2 required you to specify a p95 against.

Caps are structural. A maximum output length is not a stylistic setting; it bounds duration, which bounds concurrency, which bounds the capacity you need. Module 3 called bounding a tactic — this is where the number comes from.

For hosted inference, the limit is somebody else’s

The third property, and the one that catches teams who planned carefully against the wrong ceiling.

With hosted inference your constraint is not hardware. It is a quota — typically expressed as requests and tokens per minute — and it is administratively rather than physically bounded. That changes the work:

• Find out the actual limits and whether they are per-minute token budgets, concurrent-request caps, or both. These fail differently.
• Plan for the rejection, not just the steady state. Exceeding a quota returns an error, and that path needs defined behaviour: retry with backoff, degrade a rung down the Module 3 ladder, or queue.
• Negotiate quota ahead of launch. It is a lead-time item with a human in the loop, not a slider.

Self-hosting inverts this: the ceiling is yours and physical, so the planning is familiar, and the trade-off is that running out means buying hardware rather than filing a request.

Batching, and what it trades

Batching processes several requests together, raising throughput substantially on self-hosted accelerators. Its cost is per-request latency: a request may wait for the batch to fill.

That is a direct trade against the latency budget from Module 2 — time to first token, specifically, which is the number users actually judge. Batching is therefore excellent for work with no human waiting (index backfill, evaluation runs, bulk classification) and a poor fit for the interactive path.

Streaming interacts here too. A streamed response holds a connection open for the full generation, so your concurrency figure is also a count of open connections, and any intermediary with its own connection limits becomes part of the capacity picture.

One multiplication, and the real ceiling

About 3,192 deflected interactions a month. Spread over 30 days that is roughly 106 a day; across three time zones with a realistically wide 16-hour active window, about 6.6 an hour, or 0.0018 requests per second.

At the Module 2 budget of 8 seconds p95 per answer:

0.0018 req/s × 8 s ≈ 0.015 concurrent requests

Even a twenty-fold burst — a product launch, an outage driving a spike in “how do I…” questions — reaches about 0.3 concurrent. One modest serving path handles this with room to spare, and no partitioning, queue-based load levelling or elastic fleet is justified by throughput.

So what actually constrains capacity here? Two things, neither of them volume.

The provider’s burst quota during a spike, because a support surge is correlated — everyone asks about the same broken thing at once. The response is already specified: the 10-second timeout with handoff to a human, which is rung four of the degradation ladder.

And the agent path’s duration, which is a different workload from answering. Module 2 budgeted 20 seconds with visible progress for an action like rescheduling a job; at that duration, even modest concurrency holds connections open far longer, so the action path’s capacity picture is not the retrieval path’s. This is one more argument for the Module 3 decision to make them separate deployables.

The planning lesson: the arithmetic took a minute and ruled out most of the infrastructure someone would otherwise have proposed.

Feeds the workbook

Your residency-compliant serving topology needs the concurrency figure and the limit that binds — quota or hardware. Write the multiplication in the document; it is one line and it justifies everything you chose not to build.

Check your understanding

1. Compute concurrency for the action path at 20 seconds per interaction. How does it compare with the retrieval path, and what does that imply?
2. Your provider quota is exceeded during a correlated spike. Which rung of the degradation ladder do you land on, and what does the user see?
3. Batching would cut your self-hosted cost per interaction substantially. Which Module 2 requirement does it threaten, and where is batching still safe?

---

→ Capacity is settled by a multiplication. Retrieval quality is settled by a decision most teams make by accident. — Chunking as an architectural decision