Hosted or Self-Hosted

Overview

Section 1 decided what does the thinking. This section decides where it runs, what that costs, and whether the data going into it is any good.

The hosted-versus-self-hosted question is usually framed as cost and control. In a regulated setting it is decided by a constraint instead — but not in the way most people assume, and the assumption is worth dismantling.

The trade-off, stated honestly

Hosted inference gives you capability without operations. Someone else runs the accelerators, handles availability, and ships improvements you inherit. You pay per token, your throughput ceiling is a quota rather than a hardware limit, and the model may change underneath you unless you can pin a version.

Self-hosted inference gives you control. Your data never leaves infrastructure you operate, the model version is frozen until you move it, and at high steady utilisation the economics can favour you. You pay for the accelerator whether or not you are using it, you own availability, and you are limited to open-weight capability — which trails the frontier by a margin that varies over time.

The decisive factor is frequently neither cost nor control. It is whether you have the operational capability, and Module 3 was blunt about this: an option you cannot staff is not an option. GPU operations — capacity planning, driver and runtime management, serving-stack tuning, availability engineering for a stateful workload — is a specialism. An organisation without it does not acquire it on the side of an initiative that already has no spare capacity.

The residency assumption is usually wrong

Here is where teams go wrong, and it costs quarters.

The reasoning goes: our data cannot leave the region, therefore we must self-host. That inference is not sound. Hosted inference can commonly be pinned to a geography — providers expose a parameter for specifying where inference runs, report back where it actually ran, and offer contractual terms covering it.

So the compliance question is not one question. It is three, and they have different answers:

graph TD
  Q1{"Can inference be<br/>pinned to the<br/>required region?"} -->|no| S["Self-host, or<br/>no feature in<br/>that region"]
  Q1 -->|yes| Q2{"Do the retention<br/>terms satisfy your<br/>obligations?"}
  Q2 -->|no| S
  Q2 -->|yes| Q3{"Can you evidence<br/>both to an auditor?"}
  Q3 -->|no| S
  Q3 -->|yes| H["Hosted, with the<br/>guarantee in the<br/>contract and the<br/>evidence in the logs"]

  classDef gate fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef self fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  classDef host fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  class Q1,Q2,Q3 gate
  class S self
  class H host

Three gates, not one. A “yes” at all three makes hosted inference compliant; a “no” at any of them forces the expensive path.

Where inference happens. Can it be pinned, and does the provider report where it actually ran? The reporting matters as much as the pinning — an assurance you cannot verify is not evidence.

What is retained, and for how long. Separate from location. Some capable models are unavailable under zero-retention arrangements, which means the strongest candidate may be ruled out by a retention term rather than by anything technical. This is the criterion that surprises people, because it appears on no benchmark and in no architecture diagram.

Whether you can evidence it. Your compliance owner has to demonstrate this to an auditor. That makes it an architectural requirement: the topology must produce per-request evidence of where processing occurred, retained long enough to satisfy the audit cycle.

Answer all three before concluding anything about self-hosting. The expensive mistake is acquiring a GPU operations capability to solve a problem a contract clause and a request parameter would have solved.

What self-hosting still buys

It remains the right answer in real situations, and this is not an argument against it.

Genuinely air-gapped environments. Jurisdictions where no provider offers in-region processing. Workloads at high steady utilisation where the arithmetic in the next topic favours you. Cases where version pinning is a hard requirement because re-validation is expensive — regulated model-risk settings, for example. And cases where the task is narrow enough that open-weight capability is simply sufficient, which is more common than frontier marketing suggests.

The cost to accept with clear eyes: you now own availability for a stateful, accelerator-bound workload, and you inherit the capability gap for as long as you stay on it.

Three gates, three answers

The constraint: several enterprise contracts forbid customer data leaving its region entirely, with GDPR and SOC 2 Type II on top. Primary region is eu-central-1, secondary us-east-1.

Gate 1 — can inference be pinned? Yes, if the provider supports geographic pinning and reports where inference ran. This becomes a requirement on the provider selection, and a veto column in the Module 4 criteria matrix rather than a scored row.

Gate 2 — do the retention terms satisfy the obligations? This is the one to settle with Lena Fischer in writing, early. It may eliminate an otherwise leading model, and discovering that after building is the expensive order of operations.

Gate 3 — can it be evidenced? The serving layer must record, per request, which region processed it and under which retention terms, retained for the SOC 2 audit cycle. That is an architectural requirement, and it belongs in the reference architecture rather than in a compliance spreadsheet.

Self-hosting: rejected on capacity, not on cost. There is no GPU operations capability among 140 engineers already fully consumed by the strangler migration, and Module 3’s displacement statement already spends the single available slot. Acquiring that capability would displace the migration itself, which Tomas Beck will not agree to and should not.

Decision: hosted, in-region, with the guarantee contractual and the evidence in the logs. That is Module 3’s bet 1 restated with its mechanism — and its reversal condition stands: if in-region processing cannot be contractually guaranteed, the feature does not exist in that region.

Feeds the workbook

This is the foundation of your residency-compliant serving topology. The rubric requires that the residency constraint visibly shapes the topology rather than being noted and ignored — the three gates are how you show that, and the third gate is the one that puts a requirement on the architecture rather than on a contract.

Check your understanding

1. Which of the three gates is unanswered for the case system, and who would you ask first?
2. An engineer argues that residency forces self-hosting. What do you ask them to check before agreeing?
3. Your provider reports the processing region per request. What do you do with that data, and for how long do you keep it?

---

→ Hosted or self-hosted, the cost structure is unlike an ordinary service. Here is the arithmetic for both. — Inference economics