The Golden Dataset

Overview

An eval is a case set, a grader and a number. This topic is the case set — the part that is expensive, unglamorous, and where most broken evals are broken.

Two ideas here are worth more than the rest combined: knowing where your expected answers came from, and covering both directions of every behaviour you care about.

Where cases come from

In descending order of value:

Real traffic. Actual questions, with their real distribution of phrasing, ambiguity and typos. Nothing synthesised matches it.

Human resolutions from handoffs. Module 5 specified capturing what the person did when the system gave up. That is a case and a high-quality expected answer, produced by an expert on a case the system demonstrably could not handle — which makes it the most valuable material available, and it accumulates for free once the capture exists.

Stakeholder-supplied hard cases. The people who handle the work know which questions are traps. Asking them is cheap and they are usually pleased to be asked.

Synthesised edge cases. For coverage of things that have not happened yet — a new feature, an attack pattern, a refusal class. Useful, and the weakest source, because synthesised cases tend to be easier and more uniform than real ones.

Where the answers come from — and why it matters

The question almost nobody asks, and it can invalidate an entire eval:

Record, per case, whether the expected answer was human-written, human-verified, or generated by a model — and which model.

Here is the failure. If your expected answers were produced by a model, then scoring by similarity to them rewards imitating that model rather than being correct. In a migration this is actively perverse: gold derived from the incumbent makes the incumbent win by construction and penalises a successor for every stylistic difference, including improvements.

The rule that follows is absolute: never use model A’s output as gold when the question is A versus B.

If model-generated answers are all you have, two honest options: have a human verify a sample before trusting them, or score with a rubric or pairwise comparison rather than reference similarity. Both are more work than pretending the provenance does not matter, and both produce numbers you can act on.

Coverage: both directions, always

graph TD
  A["Eval only contains<br/>'should answer' cases"] --> B["A system that always<br/>answers scores <b>100%</b>"]
  B --> C["Optimisation drives<br/>toward always answering"]
  C --> D["Confident errors rise,<br/>and the eval cannot see it"]

  classDef start fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef bad fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class A start
  class B,C,D bad

A one-sided case set produces one-sided optimisation. The fix is cheap and almost always omitted: include the cases where the correct behaviour is to decline.

The general rule: for every behaviour you evaluate, include its negative. “Does it answer when it can” needs “does it decline when it should”. “Does it escalate when required” needs “does it refrain from escalating unnecessarily”. “Does it use the tool” needs “does it avoid the tool when it should not”.

This connects directly to Module 2. The unnecessary-handoff ceiling exists because a system that declines everything satisfies an accuracy floor perfectly. The both-directions rule is the same insight applied to the measuring instrument rather than to the requirement.

Hygiene checks worth running once

A short list that catches the most damaging problems, applied to a sample rather than the whole set:

Re-derive ten expected answers independently. Widely used benchmarks carry meaningful label error; your hand-built set will too. Wrong labels cap measurable accuracy for reasons that have nothing to do with the system.
Look for label leakage. Does the expected answer, or a paraphrase, appear anywhere the system can see — the prompt, an example, a document it can retrieve? This is common in sets assembled by copy-pasting.
Check whether cases are answerable from memory. If the point is to test retrieval, a question about a well-known fact can be answered without retrieving anything, and the case measures nothing.
Check for a trivial baseline. Could a system that always says one thing score well? If a no-op scores far above chance, part of what you are measuring is an artifact.
Check difficulty headroom. If the baseline already scores near the ceiling, the set cannot discriminate between candidates, and a tuning exercise over it will move cost rather than quality.

Ownership and refresh

A golden set with no owner decays into a confident measurement of last year’s product, which is worse than no measurement at all because it is green.

Three assignments make it live:

An owner — the person accountable for the set reflecting what the business cares about. Not engineering; whoever owns the outcome.
A refresh cadence, dated, with the set and the grader versioned together. Scores from before and after a grader change are not comparable, and a set with no version makes that undetectable.
A feed from production. Every new failure mode found in reality becomes a case. Saturated items get replaced with harder ones. This is the mechanism that keeps the set measuring the present.

The cost is real: curation is domain-expert time, repeatedly, from people with other jobs. Module 3’s displacement discipline applies — name whose time it is.

The 300-question set, specified

Module 2 asked for ≥ 90% correct on 300 curated questions, with confident errors ≤ 1% and unnecessary handoffs ≤ 15%. Here is what the set has to contain for those numbers to mean anything.

Sources. Real tickets drawn from the 4,560 the audit called answerable, sampled across topics rather than cherry-picked. Hard cases supplied by Grace Adeyemi’s team, who have seen 12,000 tickets a month and know the traps. And a growing feed of human resolutions captured at handoff, per Module 5’s protocol.

Gold provenance, tagged per case. Documentation-derived answers are human-verified against the current docs. Handoff resolutions are human-written by the agent who resolved them — the highest-quality labels in the set. No expected answer is model-generated, which matters because Module 4’s provider decision carries a reversal condition: the day a provider swap is evaluated, model-derived gold would make the incumbent win by construction.

Both directions, as four groups:

Group	Correct behaviour	Why it must be present
Documented questions	Answer, grounded, with a citation	The deflection case
Out-of-scope questions — contracts, pricing	Decline	Module 2’s refusal boundaries; without these, always-answering scores 100%
Ambiguous questions	Ask a clarifying question	Neither answering nor declining is right
Adversarial cases	Refuse, and never treat retrieved content as instruction	The safety section’s suite, scored in the same run

Ownership. Maya Ortiz owns the set, because she owns the deflection outcome. Grace Adeyemi’s team supplies and verifies cases, which is a named capacity cost rather than a favour. Refresh quarterly, plus continuous addition of production failure modes, with set and grader versioned together.

The honest gap today: none of this exists, and the first version will be assembled from the audit sample rather than from production traffic the platform has not yet seen. That is fine, and it should be labelled — a set built before launch measures what you expected, not what arrives.

Feeds the workbook

The rubric requires golden-dataset ownership and refresh cadence to be assigned. Add gold provenance per case and the both-directions groups — those two are what make the set capable of detecting the failures you actually care about.

→ The set exists. Something has to score it, and the obvious choice has a list of specific biases. — Graders and judges