Threat Modelling an LLM System

Overview

The cost pillar fails quietly by drifting. This pillar fails quietly in a worse way: a successful attack produces a 200 response and a fluent answer, and nothing in your monitoring distinguishes it from a success.

Threat modelling itself is prerequisite material — assets, entry points, actors, impact, controls. What this topic covers is the three things that are genuinely different here, because each one invalidates an assumption the standard method relies on.

Three differences that matter

Everything the system reads is an input. In a conventional system, untrusted input arrives through request parameters, and you know where they are. Here the attack surface is the union of everything that reaches the model’s context: the user’s message, retrieved documents, historical records, tool results, file contents, and the descriptions of the tools themselves. Module 5 made the point about tool descriptions; the general rule is broader and worth stating plainly — if it enters the context, it is an input.

The boundary between instruction and data is a convention, not a mechanism. A conventional system separates code from data structurally; a database driver distinguishes a query from a parameter because something enforces it. A prompt is one undifferentiated stream of text. “This part is your instructions, this part is content to consider” is a request, not a guarantee, and there is no parser to escape to. That single fact is the root cause of the next topic.

Actions give the attack a price. Module 5’s action boundary established which operations the system can perform. An assistant that can only answer has an attack surface measured in embarrassment; one that can issue a credit has an attack surface measured in currency. This is uncomfortable and also useful — it is the rare security argument that can be made in a budget meeting.

The asset inventory

graph LR
  A["User message"] --> C["One undifferentiated<br/>context"]
  B["Retrieved documents<br/>and records"] --> C
  D["Tool results"] --> C
  E["Tool descriptions"] --> C
  C --> F["Actions · data · credentials<br/>· budget · traces"]

  classDef known fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef forgotten fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  classDef ctx fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef assets fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  class A known
  class B,D,E forgotten
  class C ctx
  class F assets

Four entry points, one of which everybody models and three of which most people forget. They converge into a single stream with no enforced boundary, and what sits behind it is the asset list.

The assets worth enumerating, because the control for each lives in a different place:

• The actions it can take, ranked by blast radius. Module 5’s tiers are the inventory.
• The data it can read, per tenant. The isolation question.
• The credentials it holds — and whether the model can reach them.
• The corpus it indexes, which the next topic argues is more exposed than it looks.
• The traces it stores, which contain everything the interaction touched.
• The budget it can spend. An attacker who can trigger expensive loops costs you money without breaching anything — a denial-of-wallet attack, and Section 1’s step cap is the control.

The actors, including an unusual one

The ordinary external attacker. The merely curious customer, who finds things by accident and tells people. The insider with legitimate access.

And one specific to this architecture: the past customer whose text is now in your corpus. They may have written it years ago with no malicious intent, or with every intent, and they are not present when it is retrieved. They have write access to your retrieval surface and no ongoing relationship with the system that reads it. The next topic is about that.

What the output looks like

A threat model here is a table, not a document: asset, entry point, actor, impact, control, and where the control is enforced. That last column is what distinguishes a threat model from a worry list, and it is the column that reveals controls which exist only as instructions in a prompt.

Two published catalogues are worth reading alongside your own: the OWASP Top 10 for LLM Applications, which enumerates the attack classes, and the NIST AI Risk Management Framework, which is heavier and useful mainly for the vocabulary a compliance reviewer may already be using. Neither substitutes for modelling your own actions and data.

The inventory, with the uncomfortable rows

Asset	Entry point	Impact	Enforced where
issue_credit	Any content reaching the context	Money leaves, repeatedly, at machine speed	Action layer: limit as configuration, confirmation gate
reschedule_job	Same	Technician sent to the wrong site	Action layer: staleness check, notification
Other customers’ tickets	Retrieval	Cross-tenant disclosure; a reportable breach	Retrieval layer: tenant scope from the session
Personal data in the corpus	Retrieval, then the response	Names, addresses, payment references disclosed	Redaction before embedding; output check
Credentials	Tool results, error messages, traces	Lateral movement	Substitution at the boundary; never in context
The $8,000 monthly budget	Any path that triggers a loop	Denial of wallet; degradation for everyone	Step cap of 12; spend cap outside the application
Traces	The trace store	Everything the interaction touched, in one place	Access control; retention; erasure path

The entry points, ranked by how often they are forgotten: the customer’s message (modelled by everyone); retrieved documentation (reviewed content, lower risk); historical ticket text (written by anyone who ever contacted support — the subject of two topics’ time); and tool descriptions, which Module 5 established arrive with system authority.

The denial-of-wallet row deserves attention because it is the one with no human attacker required. A question that reliably triggers a long loop, asked repeatedly, degrades the service for everyone once the $6,400 threshold moves the platform to its cheaper configuration. The control already exists — Module 5’s step cap and Section 1’s degradation — but nobody wrote it down as a security control, and so nobody would defend it under deadline pressure.

That is the value of the exercise: it renames existing engineering decisions as the controls they actually are, which is what stops them being removed by someone optimising for something else.

Feeds the workbook

This is the skeleton of your threat model. Build the table with the enforcement column, and include the three entry points that are not the user’s message — the rubric specifically requires indirect injection through ticket content, which is the next two topics.

Check your understanding

1. List everything that reaches your system’s context. Which items had you not thought of as inputs?
2. Pick one control in your design that exists only as an instruction in a prompt. Where should it be enforced instead?
3. Describe a denial-of-wallet attack on the case system, and name the control that already mitigates it.

---

→ Three of the four entry points carry the same attack. It is the central one, and it has no complete defence. — Prompt injection