Safety and Alignment Requirements

Overview

Module 1 established guardrails as structure rather than configuration. This topic writes them as requirements — which means attaching numbers, naming the content you do not trust, and deciding what gets enforced outside the model.

Module 6 does the full threat model. What belongs here is the subset that has to exist before a structure is chosen, because it determines where boundaries sit.

Three families, and the one people forget

Injection resistance. Can instructions smuggled into content override the system’s own? Everyone remembers to consider what a user types. Far fewer consider what arrives through retrieval.

Output boundaries. Can the system be made to say something it must not — another customer’s data, a policy that does not exist, a personal detail lifted from an old record?

Action authority. When the system can act, which actions are permitted, under whose authority, and enforced where?

The second-order version of the first family is the one that gets missed, and it is the most architectural of the three.

Trust labels on content, as a requirement

A retrieval system assembles a request out of text from several places. Those places do not deserve equal trust, and nothing in the assembled request records which is which unless the architecture makes it so.

graph LR
  SI["System instructions<br/><i>trusted</i>"] --> AS["Assembled<br/>request"]
  DO["Product documentation<br/><i>reviewed, semi-trusted</i>"] --> AS
  HT["Historical ticket text<br/><i>untrusted — written<br/>by anyone</i>"] --> AS
  UI["Live user input<br/><i>untrusted</i>"] --> AS
  AS --> MO["Model"] --> ACT["Actions with<br/>real effects"]

  classDef trusted fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef semi fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef untrusted fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  classDef plain fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  class SI trusted
  class DO semi
  class HT,UI untrusted
  class AS,MO plain
  class ACT untrusted

Four content sources at three trust levels, flowing into one request. The requirement is that the trust level survives the assembly — because by the time the model sees the text, the distinction is gone unless you preserved it.

The requirement that falls out of this is specific and testable: every piece of content entering a request carries a trust label, and content below a stated trust level can never be interpreted as instruction. Writing it forces a structural element — the retrieval layer must track provenance, not just relevance — which is exactly the kind of thing that is cheap now and a rewrite later.

Authority is enforced outside the model, or not at all

The most common safety requirement in draft documents is also the weakest:

“The assistant must not issue a credit above the approved limit.”

As written, that is a hope. It asks a probabilistic component to reliably refuse something, and the whole of Module 1’s Section 2 explains why that is not a guarantee. Anyone who can phrase a request persuasively is testing a boundary that has no enforcement behind it.

The correct form moves the enforcement:

“The action layer rejects any credit above the approved limit, regardless of what the model requests. The limit is configuration in the action layer, not text in a prompt.”

Same intent, completely different structure. One is a sentence in a prompt; the other is a component with a policy, an audit record and a test. A safety requirement that can only be satisfied by the model behaving well is not a requirement. Each action tier from the blast-radius model needs its enforcement point named this way.

Making safety measurable

Safety requirements default to adjectives even more readily than performance ones, so they need the same treatment as accuracy: a named set, a judge, and a number.

• An adversarial suite, maintained like the golden set: known injection patterns, attempts to extract other customers’ data, attempts to escalate an action beyond its tier. The requirement is a pass rate on that suite, run on every release.
• Leakage counts, measured on real traffic samples rather than only on the suite.
• A refusal-appropriateness rate, because an over-blocking system is the failure mode nobody writes a ticket about.

State the uncertainty plainly: passing an adversarial suite is evidence of resistance to known attacks, not proof of resistance. Treat the number as a floor that must not regress, not as a safety case. That honesty matters when you present it — a compliance owner who hears “we are secure” stops trusting you, and one who hears “here is what we test, here is what we do not yet cover” starts.

The cost, and who decides it

Checks add latency and spend money per call, and both were budgeted in the two previous topics. The third cost is the one that kills adoption: a personal-data filter in a support tool will flag legitimate requests constantly, because support tickets are made of customer names and addresses.

That trade-off is not yours to resolve alone. The person who can block on compliance owns how strict; the team who will use the tool owns how liveable. Your job is to make both numbers explicit in the same table so the trade is visible rather than settled by whoever tunes the filter last.

Safety requirements with enforcement points named

Requirement	Measure	Enforced where	Owner
Content below “reviewed” trust level is never interpreted as instruction	0 successful injections on a 200-case suite, run per release	Retrieval layer: provenance label on every chunk	Lena Fischer
No personal data from historical tickets appears in an answer	0 leaks on a 500-case suite; monthly sample of 200 live conversations	Output check, plus redaction before embedding	Lena Fischer
No cross-customer data access	0 on suite; tenancy assertion on every retrieval	Retrieval layer, enforced by query scope — not by prompt	Lena Fischer
Credits above the approved limit are impossible	Attempted-and-rejected events logged and reviewed weekly	Action layer policy; limit is configuration	Dev Raghunathan
Refusals are appropriate	≤ 5% of legitimate requests blocked, measured on Grace Adeyemi’s team’s sample	Guardrail tuning, reviewed with support	Grace Adeyemi

Three things this table does that a prose safety section does not. It names where each rule is enforced, so none of them rest on the model’s good behaviour. It gives the refusal rate an owner who has an interest in it being low — the support lead, not the compliance lead — so the over-blocking failure has a defender. And because the tickets contain names, site addresses, phone numbers and occasional payment references, the redaction requirement lands before embeddings exist rather than after, which is the difference between a pipeline stage and a re-indexing project.

Feeds the workbook

Your ASR list should carry the trust-label requirement and the action-authority requirements, with “forces a structural element” as the stated reason for significance. The refusal-rate ceiling belongs in the NFR budget.

Check your understanding

1. Take the weak form — “the assistant must not reveal another customer’s data” — and rewrite it so that satisfying it does not depend on the model behaving well.
2. Which content source in your own system would you have labelled trusted without thinking about it? What is actually in it?
3. Your refusal rate is 5% and compliance wants it lower. What do you need to know before agreeing, and whose number is it?

---

→ Five requirement types, all with numbers, all pulling against each other. Time to put them in one place and admit that. — The NFR budget