Graders and Judges
Overview
Section titled “Overview”The case set exists. Something must turn each output into a score, and that something is where a surprising share of eval problems live.
The ordering principle mirrors the guardrails topic: use a model to grade only what cannot be checked any other way.
Grade outcomes, not paths
Section titled “Grade outcomes, not paths”Before choosing a grader type, one rule that applies to all of them.
A grader should reward reaching the right result, not taking a particular route. Requiring an exact tool sequence, a specific phrasing, or a prescribed intermediate step fails a system that solved the problem a different valid way — and it quietly optimises your system toward imitating whatever the case author happened to do.
For anything that acts on the world, this sharpens into a specific instruction: grade the end state, not the transcript. Run the task against a disposable environment and then check what it left behind — was the job actually moved to the right slot, was the credit the right amount against the right ticket, was anything touched that should not have been, were the step and cost budgets respected.
A judge reading a transcript is grading the narration. The environment is the answer.
Grader types, cheapest and most reliable first
Section titled “Grader types, cheapest and most reliable first”graph TD A["<b>1 · Structural checks</b><br/>schema valid, citation present,<br/>declined or not<br/><i>free, certain</i>"] --> B["<b>2 · End-state checks</b><br/>did the world change correctly<br/><i>cheap, decisive</i>"] B --> C["<b>3 · Reference comparison</b><br/>against a human-written answer<br/><i>limited by phrasing</i>"] C --> D["<b>4 · Human rubric</b><br/>expensive, the calibration source"] D --> E["<b>5 · Model as judge</b><br/>scalable, biased,<br/><i>usable only once calibrated</i>"] classDef free fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff classDef mid fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff classDef human fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff classDef judge fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff class A,B free class C mid class D human class E judge
Five grader types. Each case should be scored by the cheapest type that can actually answer the question — and a great deal more is structurally checkable than teams assume.
A great deal is structurally checkable. Did the answer cite a source? Is the cited source one that was actually retrieved? Did the system decline, when declining was correct? Is the output schema valid? Was the credit amount within the limit? None of those needs a model, and all of them are properties Module 2 asked for.
Retrieval quality deserves its own grading, separately from answer quality, because an answer can be wrong for two very different reasons. The standard framing — faithfulness of the answer to the retrieved context, relevance of the answer to the question, and precision and recall of the retrieved context itself — is what the RAGAS family of metrics formalises. Worth knowing, with one caveat: those metrics are themselves usually computed by a model, so they inherit everything in the next section.
When the grader is a model
Section titled “When the grader is a model”Model-as-judge is scalable and sometimes the only option for genuinely semantic properties. It also has a specific, documented list of biases, and each has a specific countermeasure.
- Position bias. In pairwise comparison, judges favour one position. Randomise which output is shown first, or score both orders and average.
- Verbosity bias. Judges reliably prefer longer answers. Tell the judge not to reward length for its own sake.
- Self-preference. A judge from the same model family tends to prefer output resembling its own. Never use the model under test as its own judge, and for close calls prefer a different family or a small panel.
- Label deference. Do not tell the judge which output is the reference, the baseline, or the human one. It will defer to the label.
- Vibes instead of a rubric. “Which is better?” produces noise. Score specific checkable properties, with structured output so parsing is deterministic.
- Injection into the judge. The judge is a model reading text that an attacker may have influenced. Treat candidate output as untrusted data, not as instructions — the whole of the previous section applies to your evaluation harness too, which is easy to forget.
- Non-determinism. Run the grader twice on the same output. If the score changes, you have grader variance stacked on top of model variance, and both need reporting.
The calibration gate
Section titled “The calibration gate”The rule that decides whether a judge’s numbers can steer anything:
Validate the judge against a few dozen cases a human labelled independently, and report the agreement. Well below about 90% on clear-cut cases means the judge prompt needs another iteration before its scores are used for decisions.
This is also what makes a judge-graded metric acceptable to a cautious stakeholder. “A model scores it” invites scepticism. “A model scores it, and it agrees with our team’s own labels 94% of the time on a sample they labelled blind” is evidence.
And test the judge on known negatives before trusting it: an empty response, an “I don’t know”, and a confident answer to a different question. A judge that passes any of those three is too lenient to use.
Two structural choices
Section titled “Two structural choices”Score atomic properties, not a blended quality number. {grounded, correct, appropriately-declined, within-policy} as separate metrics is more reproducible, easier to calibrate, and diagnostic — when a number moves you know which property moved. A single “quality” score tells you something changed and nothing more.
Make the aggregation match the question. A mean is right for typical-case quality. For rare high-stakes behaviours it is actively misleading: an irreversible wrong action diluted across 300 cases disappears into the average. For those, use worst-case or fail-on-any — one occurrence is a failure of the run.
Prove the harness before trusting it
Section titled “Prove the harness before trusting it”Two runs, minutes each, that catch most wiring bugs:
An oracle run — feed the expected answers through the grader. They should nearly all pass. If they do not, the harness or grader is broken, not the model.
A null run — feed empty output, or a constant answer. It should fail almost everything. If it does not, the grader is too lenient and every number it has produced is suspect.
Feeds the workbook
Section titled “Feeds the workbook”Your evaluation plan should name the grader type per property and the aggregation per property. The fail-on-any rows are where the rubric’s “what worse means numerically” starts to bite, since one occurrence is a different kind of trigger from a moving average.
→ Graders produce numbers. The last question is whether those numbers can see the change you are looking for. — Regression and drift