Module 6: Start Here

Overview

Modules 4 and 5 designed a system. This module is about running it.

What does it take to operate an AI system in production without it bankrupting you, leaking data, or quietly getting worse?

Three pillars — cost, safety, evaluation — and they are one module rather than three for a reason worth stating up front.

Why these three belong together

All three fail quietly.

A cost problem does not page anyone. Spend drifts up over weeks, each individual request looks reasonable, and the finding arrives as an invoice.

A safety failure does not throw. A system talked into revealing something, or into taking an action it should have refused, returns a 200 and a fluent answer. Nothing in your monitoring distinguishes it from a success.

A quality regression does not crash. The model provider updates, the documentation drifts, customer questions change, and the system answers slightly worse every week while every dashboard stays green.

Module 1 named this as the characteristic failure mode of probabilistic systems. This module is the engineering response: in all three cases, the only defence is a measurement you deliberately built, because the system will not tell you.

This is also the module that decides whether the initiative becomes the second entry in the scar-tissue file. The 2023 feature was switched off because it degraded quietly and nobody was measuring — every topic here is, in some sense, about not repeating that.

How the module is built

Three sections, matching the three pillars. This is the heaviest module in Part 3.

graph TD
  subgraph S1["Section 1 — Unit economics and cost"]
    A["1 · Pricing models"]
    B["2 · The unit<br/>that matters"]
    C["3 · Why loops cost<br/>what they do"]
    D["4 · Cost levers,<br/>in order"]
    E["5 · Cost governance"]
    A --> B --> C --> D --> E
  end
  subgraph S2["Section 2 — Safety, security and privacy"]
    F["6 · Threat modelling<br/>an LLM system"]
    G["7 · Prompt injection"]
    H["8 · Poisoning and the<br/>writable corpus"]
    I["9 · Layered<br/>guardrails"]
    J["10 · Privacy<br/>by design"]
    F --> G --> H --> I --> J
  end
  subgraph S3["Section 3 — Evaluation"]
    K["11 · From tests<br/>to evals"]
    L["12 · The golden<br/>dataset"]
    M["13 · Graders<br/>and judges"]
    N["14 · Regression<br/>and drift"]
    K --> L --> M --> N
  end
  E ==> F
  J ==> K
  N ==> O(["Workbook — Exercise 6"])

  classDef s1 fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef s2 fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  classDef s3 fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef cap fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  class A,B,C,D,E s1
  class F,G,H,I,J s2
  class K,L,M,N s3
  class O cap

Three pillars, one failure mode. Each section ends where the next begins: cost control changes what you can afford to check, safety determines what must be checked, and evaluation is how you know any of it still works.

Section 1 — Unit economics and cost

1. Pricing modelsPer token, per hour, provisioned, batched — and what each implies structurally.

2. The unit that mattersCost per completed task, the denominator, and why you price the tail.

3. Why loops cost what they doAgent cost grows with the square of turn count. Here is the arithmetic.

4. Cost levers, in orderFree wins before trade-offs, and the one everyone mistakes for a saving.

5. Cost governanceAttribution, budgets, hard stops, and who owns the number.

Section 2 — Safety, security and privacy

6. Threat modelling an LLM systemAssets, entry points, and why content is part of the attack surface.

7. Prompt injectionDirect and indirect — and why the indirect kind is the architectural one.

8. Poisoning and the writable corpusWhen your retrieval corpus is something strangers can write to.

9. Layered guardrailsDefence in depth, what each layer costs, and how to measure it.

10. Privacy by designErasure that reaches derived data, and a design a compliance owner can approve.

Section 3 — Evaluation

11. From tests to evalsWhat replaces unit-test thinking, and what unit tests still do.

12. The golden datasetWhere the cases come from, where the answers come from, and who owns it.

13. Graders and judgesMetrics, model-as-judge failure modes, and calibrating against humans.

14. Regression and driftWhat 'worse' means numerically — and whether your eval can even see it.

How to work this module

About seven hours — the heaviest in this Part:

Read the fourteen topics — about 4 hours. Section 3 is the one that corrects a number you wrote in Module 2, so do not skip it because evaluation sounds like testing.
Optional — Workbook exercise 6 — about 3 hours. Four artifacts, one of which may disprove the business case.

In the workbook

Four artifacts for Lintel:

A unit-economics model proving — or disproving — the $3.00-per-ticket target, with the levers that get you there
A threat model with layered guardrails, focused on the agent’s action surface
A privacy design satisfying GDPR, residency and SOC 2, written to be reviewed by the person who can block you
An evaluation plan with a golden-dataset specification and named regression criteria

Before you start

Have Module 2’s NFR budget open. The cost ceiling, the accuracy floor and the regression trigger all came from there, and this module tests two of the three against arithmetic.

Numbers quoted in this module are directional, not guarantees. Where a measured result from published work is cited, it is a snapshot from someone else’s workload — useful for sizing a lever before you spend anything, never a substitute for measuring your own. Every figure here should be read as “this is the order of magnitude to expect; go and verify”.

→ Start with Pricing models.