From Tests to Evals

Overview

The third pillar. Cost tells you what the system spends and safety tells you what it must not do; neither tells you whether it is any good, or whether it is still as good as it was last month.

Module 1 said testing becomes measuring. This section builds the measuring — and starts by narrowing the claim, because “you cannot unit test an AI system” is repeated far more often than it is true.

What unit tests still own

The parts of your system that are deterministic are still deterministic, and they are not a small residue. From Modules 4 and 5:

The limit check in the action layer.
The tenancy assertion.
The idempotency key composition.
The redaction stage.
The allowlist.
The step cap, the timeout, the spend cap.
The retrieval layer’s provenance labelling.

Every one of those is ordinary code with an expected output, and every one of them is a security control. Which produces a useful inversion of the usual framing:

The controls are deterministic and must be unit tested. It is only the model’s output that needs evals.

A team that reaches for evals to check whether credits above the limit are refused has misallocated the work — and built a probabilistic check for a deterministic property, which the previous section argued is a strict downgrade. Test the gate with a unit test. Evaluate the answer with an eval.

What an eval is

Three parts, and all three are required:

A frozen case set. Fixed inputs, so results are comparable across changes. Freezing is what makes it a measurement rather than an anecdote.
A grader. Something that turns an output into a score.
A reported metric with variance. A single number with no error bar is a point estimate masquerading as a fact.

The last part is the one most often missing, and the final topic in this section is largely about why it matters more than it sounds.

Four layers of checking

graph TD
  A["<b>Unit tests</b><br/>deterministic components<br/><i>fast, certain, free</i>"] --> B["<b>Offline evals</b><br/>frozen set, pre-release<br/><i>comparable across changes</i>"]
  B --> C["<b>Online signals</b><br/>real traffic, post-release<br/><i>catches what the set missed</i>"]
  C --> D["<b>Human review</b><br/>the residue<br/><i>expensive, irreplaceable</i>"]

  classDef cheap fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef mid fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef live fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef human fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class A cheap
  class B mid
  class C live
  class D human

Four layers, increasing in cost and in realism. Each catches a class of problem the layer above cannot, which is the test for whether a layer earns its place.

Offline evals answer did this change make things better or worse? They are comparable across changes because the inputs are frozen, which is their entire value — and their limitation, because they only contain what someone thought to put in them.

Online signals answer is it working in reality? Real traffic contains the question mix nobody anticipated and the drift nobody scheduled. The signals are indirect — handoff rate, repeat-question rate, how often a human rejects the agent’s proposed action — but they are measured on the thing that matters rather than on a proxy.

You need both. Offline evals gate a release; online signals tell you whether the gate was measuring the right thing. A team with only offline evals ships confidently against last quarter’s reality; a team with only online signals cannot attribute a change to a cause.

Human review is the residue and the calibration source. Expensive, and the only layer that can tell you the eval itself is wrong — which the final topic in this section argues is a surprisingly common condition.

Where evals cost money

Two costs worth planning for rather than discovering.

Each run spends real inference. A case set run against several candidate configurations is a multiple of that, and the sweeps in Section 1 are exactly this shape. Which is why Section 1 flagged evaluation runs as a batch workload: nobody is waiting, and the discount is substantial. Treat eval spend as a line in the budget rather than an unplanned draw on the ceiling.

Curation is human time from people who know the domain. That is the scarce input, not the compute, and it is the reason the next topic is about where cases and answers come from.

What gets tested how

Sorting the system by checking layer makes the allocation obvious — and reveals that most of the security work is unit tests.

Unit tests, deterministic: the credit limit check; the tenancy assertion on every retrieval; idempotency key composition; the redaction stage; the tool allowlist; the step cap of 12; the 25-second timeout; the spend cap. Every one of these is a control from the previous section, and every one has an expected output.

Offline evals: answer correctness on the 300-question golden set; grounding in retrieved documentation; appropriate declines; the adversarial suite from the safety section; and — distinct from answer quality — retrieval quality on its own, because an answer can be wrong because retrieval failed rather than because generation did.

Online signals: handoff rate against Module 2’s 15% ceiling; repeat-question rate, which detects answers that satisfied the grader and not the customer; confirmation-rejection rate, which is the rate at which Grace Adeyemi’s agents decline the agent’s proposed action — an unusually direct quality signal that costs nothing to collect because the gate already exists; and per-tier accuracy if a cheap-model cascade ships.

Human review: the residue, and the source of new cases.

The budget consequence. Eval runs against 300 questions, repeated across configurations, are a recurring cost inside the $8,000 monthly inference ceiling. Batched at roughly half price they are affordable; run synchronously and frequently during a tuning cycle they are not trivial. That connection — Section 3 depends on Section 1’s batch lever to be affordable — is worth stating in the plan rather than discovering in the invoice.

Feeds the workbook

Your evaluation plan should begin by sorting the system into these four layers. The rubric asks what “worse” means numerically and what happens when it triggers; that belongs to the last topic, but you cannot write it until you have said which layer is doing the measuring.

→ An eval is a case set, a grader and a number. The case set is where it is usually broken. — The golden dataset