Regression and Drift

Overview

The last topic of the module, and the one that goes back and corrects an earlier decision with arithmetic.

Module 2 specified a regression trigger: a three-point fall in the correctness rate blocks the next release. That sounded careful. This topic shows it cannot work as written, and what to do instead.

The noise floor

An eval score is a measurement, and measurements have a resolution. For a pass rate measured on n cases with R repetitions each, the smallest difference you can distinguish is roughly:

noise floor ≈ 1 / √(n · R)

That single expression settles whether a regression trigger is meaningful. Put the Module 2 numbers through it:

300 cases, run once: 1 / √300 ≈ 0.058 → ±5.8 points
The trigger: 3 points

The trigger is inside the noise. Two consecutive runs of an unchanged system can differ by more than three points, which means the gate will block good releases on random variation, and a genuine three-point degradation will pass unnoticed inside the same spread. It is worse than no gate, because it produces confident-looking signals in both directions.

300 cases × 4 runs — noise ±2.9 points −3 pts detectable

The grey band is the range two runs of an unchanged system can differ by. On the upper line the trigger sits inside it; on the lower line, four repetitions narrow the band until the same trigger clears it.

The fixes, cheapest first. More repetitions is the cheapest, because the case set is already curated and paired comparisons go further than independent ones. 300 cases run four times gives roughly ±2.9 points, which clears a three-point trigger. More cases works too and costs curation time. A continuous metric instead of a binary pass carries more information per case. Or widen the trigger to something the current set can actually see.

What you must not do is leave a trigger in place that the measurement cannot support. And note where this lands: four runs instead of one is four times the inference, which is precisely why Section 1 flagged eval runs as batch work at roughly half price. The statistics and the cost lever are the same conversation.

Do not score the plumbing

The other way an eval produces confident nonsense, and it is purely a harness property.

Conflation is when a non-model failure lands in the same column as a model result. A timeout, a rate-limit error after retries, an unparseable output, a response truncated at the output cap, a tool that threw, a grader that crashed — if any of those is silently scored as a failure, the headline number is measuring your infrastructure.

Three rules:

Attempts that never produced a scorable output go to an error sidecar with a failure class, never into the results.
A truncated response is recorded and shown but not averaged in as wrong.
“No answer” is not “negative answer.” A system that correctly declines must not score the same as one that crashed. Without this distinction, a harness that errors on every input scores identically to one that carefully found nothing — and refusals are a graded outcome in their own right, not an error.

Drift, and what each source requires

Quality decays for several distinct reasons, and the response differs:

Source	Signal	Response
Provider updates the model	Offline score moves with no change on your side	Pin versions where possible; re-baseline deliberately
The corpus changes	Retrieval metrics move before answer metrics	The refresh semantics from Module 4
Question mix changes	Online signals move while offline scores hold	New cases into the golden set
Your own prompt or config changes	Attributable, if one change at a time	The one-lever-per-change discipline from Section 1

The second row is worth noticing: grading retrieval separately from answers is what lets you tell a documentation problem from a model problem, which Module 4 argued is usually the likelier cause.

The gate, and the online half

A regression criterion needs three parts, or it is a dashboard: the number, the cadence, and what happens automatically. Blocking a release is a real gate. Emailing someone is monitoring theatre — the same point Module 3 made about health checks.

And offline gates cannot see everything. The online signals from earlier in this section are what catch the rest: handoff rate, repeat-question rate, and the confirmation-rejection rate, which is the proportion of proposed actions a human declines. That last one is the most direct quality signal in the system and it costs nothing to collect, because the gate already exists.

One comparability rule: version the case set and the grader together. Scores from either side of a grader change are not comparable, and without versioning nobody can tell that is why the number moved.

Correcting the Module 2 trigger

Module 2’s regression clause: measured weekly against the golden set; a three-point fall in the correctness rate blocks the next release.

The arithmetic. 300 questions run once gives a noise floor of about ±5.8 points. The three-point trigger is inside it. As written, the gate would block releases on noise and miss real three-point regressions — and because it looks rigorous, nobody would question the numbers it produced.

The correction, with its cost. Run the 300-question set four times per evaluation, giving roughly ±2.9 points and making the three-point trigger detectable. The cost is four times the inference per run, which lands inside the $8,000 monthly ceiling only because evaluation is batch work at roughly half price — the two sections of this module meeting in a single decision.

The alternative, if that spend is refused, is to widen the trigger to six points and say plainly that smaller regressions are not detectable with the current set. That is a worse outcome and an honest one; what is not acceptable is keeping a three-point trigger on a single run.

The fail-on-any rows are different. Confident errors on credits, cross-tenant disclosure and successful injections are not averages — one occurrence blocks the release. No noise-floor argument applies, because the threshold is one.

What happens when it triggers. The release is blocked, and the confidence threshold rises until the drop is explained — the behaviour Module 2 specified. Automatic, not advisory. Maya Ortiz is notified as the owner of the accuracy floor; Ivan Petrov sees the same report, because a visible, automatic regression gate is the most direct answer available to this will degrade quietly like 2023 did.

And the online half, because the weekly offline run cannot see a changing question mix: handoff rate against the 15% ceiling, repeat-question rate, and the confirmation-rejection rate from Grace Adeyemi’s team.

Feeds the workbook

Two rubric items land here: the eval plan must define what “worse” means numerically and what happens when it triggers. Run the noise-floor arithmetic for your own set size — if your trigger is inside the floor, fixing that is the single highest-value correction in this module’s workbook exercise.

→ Three pillars done — cost, safety and evaluation. One of them may disprove the business case. — Module 7 — Designing Under Uncertainty

Optional workbook: Exercise 6 turns this module into artifacts. The exercises build on each other, so work them in order or not at all.