Exercise 7: Designing Under Uncertainty

Overview

Four artifacts for Lintel, about three hours. This exercise closes Part 3.

Two of the four are reviews of your own work from Exercises 4–6, which makes this the least comfortable exercise in the guide and the most likely to change your design. That is the intent: six exercises have built an architecture, and the honest final step is to attack it.

You need your artifacts from Exercises 4, 5 and 6 in front of you. If they do not exist, write down the stack choice, the action boundary and the eval plan first — you cannot review an architecture you have not committed to.

How to work

The order matters. Do the antipattern review before the red-team, so the red-team tells you what your own review missed rather than duplicating it.

Optimisation plan — ~40 min
Uncertainty policy — ~50 min
Antipattern review of your own design — ~50 min
AI-assisted red-team — ~40 min
Self-check — six pass/fail items

Artifact 1 — Optimisation plan

One page plus a table.

Open with what you would profile and what you predict — not with a list of techniques. A plan written before a profile exists must say so and name the measurement that would change it.

The table, one row per lever: expected effect · quality cost · status. Required statuses include apply now, blocked on an evaluation, and not our lever — keep that last category in, because omitting it invites the same proposal in month three.

Cover: caching and its duration per path, output shaping, retrieval depth, reasoning effort, parallelisation and where it is unsafe, and anything you are rejecting with its reason. Note which rows are blocked until a golden set exists.

The test: every row states what it spends, not only what it saves.

Artifact 2 — Uncertainty policy

Two pages. The most load-bearing artifact in this exercise.

Per action, a row covering:

Retry category — retry freely · retry with a key · never retry blind
Idempotency key composition, issued by your code
Unknown-outcome behaviour — the state where you cannot tell whether it was applied. Verify-then-decide, and what happens when verification itself is unavailable
Confidence signal and threshold — the specific observable, not “if confidence is low”; who owns the number; how it is calibrated

Then the degradation ladder, with four columns per rung: trigger · what the system does · what the user sees · the quality bar. That last column is the one most designs omit, and it decides whether a degraded mode still deflects or merely annoys.

Finish with how each rung gets exercised, and the rung-attribution metric.

Artifact 3 — Antipattern review of your own design

Two pages, structured by the six review questions, not by the antipattern list — the questions are what find things.

Walk your Exercises 4–6 artifacts against:

Does it satisfy each stated requirement, named individually?
What did it omit?
What did it assume without saying?
Where is a trade-off unnamed?
Does it respect the constraints, or assume them away?
Would the person who can block this approve it?

Then check the six antipatterns explicitly: hallucination amplification, context overflow, the eval-free deployment, the unbounded agent, the cost spiral, and the “just add RAG” reflex. For each: present or absent, detection signal, and structural fix.

The rubric requires you to find at least one real problem. If you find none, you have reviewed the document rather than the design — go back to question two, because omissions are where real defects live.

Artifact 4 — AI-assisted red-team

One page, and it has to be actually run.

The prompt you used. Instruct the model to attack, not review, and name a specific adversarial role — the compliance owner with rejection authority, the sceptical principal engineer, the CFO asking why the economics close.
The raw findings, however many came back.
Your triage — which you accepted, which you rejected, and why. The accepted ones are your defect list: Exercise 9 turns each into a decision record. The rejections demonstrate judgement as much as the fixes do.
What it surfaced that you had missed, and your response to each.
What it could not possibly have found — the organisational realities invisible in any document. This list is the bridge into Part 4.

The test: it found something your own review did not. If it found nothing, your prompt asked for a review.

Self-check

Pass or fail.

Every optimisation states its quality cost, not only its savings.
The fallback path is fully specified — what the user sees when the model is unavailable or unsure, with a quality bar per rung.
Retry policy accounts for actions with side effects. A retried credit is a second credit; the unknown-outcome state has a designed behaviour.
The antipattern review found at least one real problem in your own design.
The red-team surfaced something you had missed, and you recorded your response.
At least one place is identified where a deterministic component should replace a probabilistic one.

If something failed

If your antipattern review found nothing, you reviewed the prose. Bring the checklist rather than deriving it from the document, and start with question two — a requirement with no mechanism is invisible to a careful reading and obvious to a list.

If the red-team produced only generic observations, the prompt asked for balance. Re-run it as a named hostile reviewer with authority to reject, and ask for the three strongest grounds.

If your uncertainty policy has no unknown-outcome column, that is the gap most likely to cost real money. A timeout on a credit is not a failure and must not be retried as one.

If no deterministic replacement surfaced, look for a decision table hiding inside a model call. Most designs have at least one, and policy checks are the usual culprit.

→ The design is complete and honestly reviewed. Next, the organisation it has to get through. — Exercise 8, after reading Handbook Module 8.