Specifying Probabilistic Outcomes

Overview

Intent says what good means. This topic puts numbers on how often the system has to achieve it, and — more importantly — on the different ways it is allowed to fail.

Module 1 established that testing becomes measuring and that somebody must own what “often enough” means. Here you write that down as a requirement.

“Accuracy” is not a requirement yet

An accuracy target is meaningless without three companions, and a target missing any of them cannot be checked:

What counts as correct. Factually right? Grounded in a source? Complete? Appropriately refused? These diverge constantly — an answer can be factually true and still wrong, because the documentation says something else and the documentation is what the company will honour.
Who judges. A human reviewer, a comparison against a reference answer, or a model asked to grade. Each has a different error profile, and Module 6 covers the failure modes of the last one.
On what set. Accuracy is a property of a measurement, not of a system. A number without a named dataset is a number somebody chose.

So an accuracy requirement reads: this share of this set, judged this way, counts as correct. Three sentences, not one adjective.

Not all errors cost the same

This is the structural insight of the topic, and it is where most specifications are lazy.

A system that declines when it should have answered has wasted a cheap opportunity. A system that answers confidently when it should have declined has done something much worse — it has produced a plausible, well-formed, wrong answer that someone will act on.

graph TD
  A["Model produces<br/>a candidate answer"] --> C{"Confident<br/>enough?"}
  C -->|yes| CR{"Actually<br/>correct?"}
  C -->|no| H["<b>Hands off</b><br/>cost: one agent's time<br/>plus a slower answer"]
  CR -->|yes| G["<b>Answered well</b><br/>the outcome you<br/>are paying for"]
  CR -->|no| W["<b>Confidently wrong</b><br/>cost: trust, and<br/>whatever the customer did<br/>with the answer"]

  classDef neutral fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef gate fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef good fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef cheap fill:#7C7C7C,stroke:#5E5E5E,stroke-width:2px,color:#ffffff
  classDef bad fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class A neutral
  class C,CR gate
  class G good
  class H cheap
  class W bad

Four outcomes, three of them acceptable. Only the terracotta one damages anything that is hard to get back — so it gets its own, much stricter number.

So specify at least two numbers, not one:

A floor on good outcomes — the share of the set answered correctly.
A ceiling on confident errors — much tighter, often an order of magnitude tighter.

And often a third: a ceiling on unnecessary handoffs, because a system that declines everything is perfectly safe and worthless. Without it, the cheapest way to satisfy the first two requirements is to answer almost nothing — which an implementation under pressure will absolutely discover.

Confidence thresholds are a business decision

“Answer if confident, otherwise hand off” needs a number, and moving that number trades the three figures above against each other. Raise it and errors fall while handoffs rise, taking the cost savings with them. Lower it and the reverse.

Two consequences for how you write the requirement.

First, the threshold belongs to whoever owns the consequences, not to you and not to the implementer. For an informational answer, that is the support owner. For an action that moves money, it is whoever owns the financial exposure — which is the same argument as the blast-radius tiers, arriving now as a requirement rather than a design note.

Second, the threshold must be a parameter, not a constant. Writing it as a requirement that will be tuned against live data forces the structure to expose it — configurable, observable, and changeable without a deploy. A threshold hardcoded in a prompt is a business decision buried where nobody can audit it.

A caution on what confidence actually means: a model’s self-reported certainty is not a calibrated probability, and treating it as one is a known trap. Specify the signal you will threshold on — retrieval score, self-assessment, agreement between two passes, or a combination — and treat its calibration as something to be measured, not assumed. Treat this as a strong recommendation rather than settled practice; the tooling here is still moving.

The requirement is not done until it says what happens on decline

A point-in-time accuracy target describes launch day. Module 1’s quiet-failure problem means launch day is not the risk.

So every probabilistic requirement needs a regression clause: how often it is measured, what counts as a meaningful drop, and what happens automatically when the drop occurs. “Measured weekly against the golden set; a three-point fall in the correctness rate blocks the next release and raises the confidence threshold until it is explained.”

That last part is what makes it architectural. An alert that emails someone is a monitoring choice. A threshold that moves automatically when quality falls is a structure — it needs a control path, a store for the current value, and a way to make the change visible.

An accuracy specification that survives scrutiny

Requirement	Value	Owner
Correct answers on the golden set	≥ 90% of 300 curated questions	Maya Ortiz
Confidently wrong answers	≤ 1% of answered questions	Maya Ortiz, with Ivan Petrov as reviewer
Unnecessary handoffs	≤ 15% of answerable questions	Grace Adeyemi
Confidence threshold	A tunable parameter, initial value set from a two-week shadow run	Maya Ortiz; Dev Raghunathan for credit actions
Measurement cadence	Weekly, plus before every release	Engineering
Regression trigger	A 3-point fall in correctness, or any rise in confident errors above 1%, blocks release	Automatic

Three things to notice. The confident-error ceiling is ten times tighter than the correctness floor, because the 2023 routing feature failed by being quietly wrong rather than by being unavailable. The handoff ceiling exists because without it the safest implementation deflects nothing and the $3.00 target becomes unreachable. And Ivan Petrov — who has no formal authority — is named as reviewer on the error ceiling, which is the elicitation move from the previous topic cashed in: he set that number, so he now defends it.

What this costs

The golden set is the expensive part: 300 curated questions with agreed answers is real work by people who have other jobs, and it must grow as the product does. Budget for an owner and a refresh cadence, or it decays into a set that measures last year’s product.

The subtler cost is that explicit thresholds remove deniability. Once “90%” is written down, the 10% is a known, accepted outcome rather than a surprise — and someone has to be willing to accept it in writing. That conversation is the requirement doing its job.

Feeds the workbook

These numbers belong in your NFR budget, and the module rubric asks for at least three AI-specific NFRs that would not exist in a deterministic system. The confident-error ceiling and the regression trigger are two of them.

→ Correctness has a number. Users experience something else entirely first: how long they waited. — Latency and perceived time