Choosing a Model

Overview

This is the decision everyone starts with and the one most often made on reputation. It is also, as the previous topic showed, a decision that constrains three layers beneath it.

What follows is not a recommendation. It is the criteria catalog — Module 3’s move applied to the fastest-moving layer in the stack.

Why benchmarks mislead

Public benchmarks measure general capability on public tasks. You have a narrow, private task, and the correlation between the two is weaker than the leaderboards imply.

Three specific failure modes:

Your task is not the benchmark’s task. A model that leads on competition mathematics tells you very little about whether it answers questions about scheduling software from your documentation without inventing a menu item.

Contamination is unfalsifiable from outside. You cannot verify that public test material was absent from training data. You can, however, verify performance on your golden set, which no provider has seen.

Aggregate scores hide the distribution you care about. A model with a slightly lower average may make dramatically fewer confident errors — and Module 2 gave confident errors their own ceiling, ten times tighter than the correctness floor, because they are the failure that ends projects.

So the primary evidence is your own evaluation set, run against each candidate. Everything below is secondary to that, and the ordering is deliberate.

The criteria that discriminate

A criterion earns a column only if serious candidates score differently on it. These usually do:

Criterion	What to look up	Why it discriminates
Quality on your golden set	Run it yourself	The only evidence about your task
Confident-error rate	Your own eval, scored separately	Ties directly to the Module 2 ceiling
Input price per million tokens	Provider’s current pricing	Retrieved context dominates input volume
Output price per million tokens	Same	Output is priced at a multiple of input — commonly several times — so answer length is a cost lever
Latency profile	Measure, don’t read	Time to first token and generation rate are separate numbers
Context window	Provider’s model metadata	Bounds retrieval depth, which bounds chunking
Reasoning-effort control	Provider docs	A capable model at reduced effort is the first cost lever, before any model swap
Caching support and semantics	Provider docs	Cache reads are roughly an order of magnitude cheaper than fresh input; writes cost slightly more than fresh input
Batch processing	Provider docs	Asynchronous work commonly carries a standing discount — typically around half price
Residency and inference geography	Provider docs and contract	Often a veto rather than a score
Data retention terms	Contract	Some capable models are unavailable under zero-retention agreements
Rate limits and quota	Provider docs	Decides whether your peak is servable at all
Exit cost	Your own assessment	Covered in the next topic but one

Two notes on using this table.

Look the numbers up, do not recall them. Prices, context windows and capability tiers change on a timescale of weeks. Providers expose model metadata through an API — context window, output cap, capability flags — which means your criteria matrix can be refreshed programmatically rather than by someone’s memory of a blog post. Treat any figure in your matrix as dated the moment you write it, and put the date next to it.

Measure latency rather than reading it. Published latency figures describe conditions that are not yours: different prompt sizes, different output lengths, different regions, different times of day. The number that belongs in your matrix is one you measured on your own prompt shape.

The two columns that eliminate rather than score

Module 3’s distinction applies with full force here. Residency and data-retention terms are usually constraints, not criteria.

A model that cannot process data in the required region, or that is unavailable under the retention terms your contracts demand, is not a low-scoring option. It is not an option. Scoring it at two out of five and letting a strong quality column carry it is precisely how a compliance boundary gets traded away by arithmetic.

This matters more than it sounds, because the strongest models are not uniformly available under every retention and residency arrangement. The best model for your task may be unavailable to you for reasons that have nothing to do with its capability — and discovering that after building against it is an expensive way to learn where the veto columns are.

Making it auditable

The deliverable is a matrix with evidence in each cell, not scores from judgement. A cell reading “4/5” is an opinion with formatting; a cell reading “91% on the 300-question golden set, measured 12 March, p95 TTFT 1.2 s at our prompt shape” is evidence.

graph LR
  E["Your golden set<br/>+ measured latency<br/>+ current pricing"] --> M["Criteria matrix<br/>evidence per cell,<br/>dated"]
  V["Residency and<br/>retention terms"] --> VG{"Veto"}
  VG --> M
  M --> D["Defensible choice<br/>+ the condition that<br/>would reverse it"]

  classDef ev fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef veto fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  classDef out fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef mat fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  class E ev
  class V,VG veto
  class M mat
  class D out

Evidence feeds the matrix; constraints bypass it and eliminate. The output is a choice plus its reversal condition, which is what makes it a bet rather than a preference.

The cost of working this way: running your golden set against several candidates costs real money and a day or two of someone’s time. That is the cheapest money in the whole project. The alternative is choosing on reputation and discovering the gap when the unit economics are already promised.

The matrix, and what it eliminates

Columns for the support platform, in evaluation order:

1. Correctness on the 300-question golden set — measured per candidate, reported with the confident-error rate separately, because Maya Ortiz’s ceiling on confident errors is ten times tighter than her correctness floor.
2. Cost per interaction at the real prompt shape — retrieved documentation dominates input volume, so this is measured with realistic context, not with a bare question. The ceiling is $1.54 at p50.
3. p95 time to first token, measured in-region at the Module 2 budget of 1.5 s.
4. Context window, which bounds how many chunks can be retrieved per answer and therefore constrains the chunking decision two topics from now.
5. Effort or reasoning control, because reducing effort on a capable model is the first cost lever to try.
6. Caching semantics, since repeat questions are common in a corpus where 38% of tickets are already documented.

Veto columns, not scored: in-region inference must be contractually guaranteed, because several enterprise contracts forbid customer data leaving its region outright. And the retention terms must satisfy SOC 2 Type II and GDPR as Lena Fischer reads them — which may eliminate an otherwise leading candidate.

Dev Raghunathan can audit every row of this. He cannot audit “we chose the best model”, and asking him to is how an architect spends credibility they will need later.

Feeds the workbook

This is the model layer of your AI stack selection. The rubric asks that model selection use documented criteria a CFO could audit rather than benchmark scores — so put the date, the measurement method, and the veto columns in the artifact itself.

Check your understanding

1. Which three criteria from the table would actually discriminate between candidates for the case system? Cut the rest and say why.
2. You cannot run a golden set yet because it does not exist. What do you do — and what do you refuse to claim in the meantime?
3. A leading candidate fails the retention terms. Write the sentence you put in the matrix, and say who needs to see it.

---

→ The matrix is built. The instinct now is to take the strongest model, and that instinct is usually wrong — though not for the reason people give. — The frontier default, and cascading