Hallucination Amplification

Overview

Four recurring failure patterns follow. This one is first because it compounds — the only antipattern here that gets worse the longer the system runs.

The mechanism is simple enough to state in a sentence: model output becomes model input. An error introduced at one step arrives at the next step as established fact.

The mechanism

A conventional error stays the size it was. This one grows.

At step one the system produces a claim that is not supported by anything retrieved. At step two that claim is in the context, indistinguishable in form from the material that was retrieved. The system reasons from it. At step three the conclusion drawn from it is also in the context. By step eight the transcript contains a small, internally consistent body of fiction, and the system is reasoning carefully about it.

The dangerous property is the divergence between two things that usually move together:

graph LR
  A["Step 1<br/>grounded<br/><i>claim appears</i>"] --> B["Step 3<br/>partly grounded<br/><i>claim reused</i>"]
  B --> C["Step 6<br/>weakly grounded<br/><i>conclusions built on it</i>"]
  C --> D["Step 9<br/>ungrounded<br/><i>internally consistent,<br/>externally false</i>"]

  classDef ok fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef warn fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef bad fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class A ok
  class B warn
  class C,D bad

Grounding falls across the loop while internal consistency rises. The system becomes more confident and less correct at the same time, which is why no confidence signal catches it.

Where it appears

Five places, and the last two are the ones nobody expects.

Agent loops. Each turn’s reasoning enters the next turn’s context. Module 5’s bound is a partial defence, because a shorter loop has less room to compound.

Summaries of summaries. Module 5 warned that a generated summary becomes what the system believes happened. Summarise a summary and the drift is multiplied with no original left to check against.

A judge from the same family as the system under test. Module 6 named this as self-preference bias; it is the same mechanism, applied to measurement rather than to output.

Generated content entering the retrieval corpus. Module 6 established that a writable corpus is an attack surface. The uncomfortable version is that you can poison your own corpus — if a resolution the system drafted gets indexed, the system’s own invention becomes retrievable fact, and now it is grounded. Permanently.

Synthetic evaluation data. Module 6’s gold-provenance rule exists for this reason: expected answers generated by a model make the eval measure imitation rather than correctness.

Detection signals

Each is cheap and specific:

• Claims that cite nothing. A structural check, already required by Module 7’s confidence topic. An answer whose assertions trace to no retrieved document is the primary signal.
• Divergence over turns. Compare late-turn claims against the retrieved set. Content appearing at step eight that was in no document at step one is the pattern.
• Confidence rising while grounding falls. If you record both — and you should — the crossing point is visible.
• Drift between a summary and its source, measured on a sample rather than assumed.

Structural fixes

Re-ground at every step rather than trusting the transcript. This is the main one, and it rests on a rule worth stating plainly: the transcript is not a source of truth. It is a record of what was said, including what was said wrongly. When a later step needs a fact, re-fetch it from the system of record rather than reading it out of the conversation.

Carry provenance through transformations. A claim should know where it came from, and a derived claim should know what it was derived from. Module 4 required provenance on retrieved chunks; this extends it forward through the loop.

Never let generated content into the corpus unreviewed. Module 6’s resolution-gating applies to your own output as much as to a stranger’s: a human decides what becomes retrievable.

Bound the loop. Fewer turns, less compounding.

The cost of all this is real: re-fetching costs latency and tokens where reading the transcript is free, and provenance tracking is plumbing nobody sees. It is the cheapest insurance in this module.

Where the loop could close on itself

The action path runs up to 12 steps and produces a plan that a human confirms. Three exposures, in increasing order of how badly they would age.

Within the loop. A claim about a customer’s entitlement, invented at step three, informs the plan at step nine. The fix is the one from the hybrid-architectures topic: entitlement is a decision table, queried, not inferred — which removes the most consequential class of invented fact entirely.

Through the handoff. Module 5’s handoff package includes what the system concluded. If an agent accepts that framing without checking the sources, the system’s error becomes the agent’s answer to the customer. The countermeasure is already in the package design: the retrieved sources travel with the conclusion so the agent can check in seconds rather than re-derive in minutes.

Through the corpus — the one that would be permanent. Module 6 proposed indexing only tickets a support agent resolved and marked useful. If the system drafts a resolution and an agent accepts it with one click, the system’s own output becomes indexed, retrievable, reviewed-looking content. Every future similar question retrieves it. A fabrication would become the canonical answer, immune to model upgrades, exactly as Module 6’s poisoning topic described — except self-inflicted.

So resolution-gating needs one more condition than Module 6 specified: a resolution drafted by the system and merely approved is not the same as one a human wrote. Either exclude machine-drafted resolutions from indexing, or require an explicit edit before they qualify. That distinction was missing from the Module 6 data strategy and it is a genuine gap this topic surfaced.

Feeds the workbook

This is the first entry in your antipattern review. The rubric asks that the review find at least one real problem in your own Modules 4–6 design — the corpus-feedback gap above is exactly the shape to look for: a mechanism that is safe against outsiders and not against yourself.

Check your understanding

1. Trace one invented fact through your own design. Where would it stop, and where would it become permanent?
2. Which of your mechanisms treats the transcript as a source of truth? What would re-grounding cost there?
3. Can machine-drafted content enter anything your system later retrieves? What approval would make that safe?

---

→ Amplification needs room to compound. The next failure is about running out of it, silently. — Context overflow and silent truncation