The Documentation Quality Problem

Overview

The module closes where it should: with the assumption underneath everything else.

Every projection of the form “a share of these requests is answerable from existing documentation” is a claim about the documentation, not about the requests. It is cheap to test and almost never tested before a quarter has been committed.

Retrieval is an amplifier with no sign of its own

Topic 5 introduced this; here is the full argument.

Over well-maintained documentation, retrieval produces grounded, current, checkable answers at a scale no support team could match. Over documentation that is thin, stale or self-contradictory, it produces answers that are confidently thin, stale or self-contradictory — faster, at greater volume, and with a citation attached that makes them look more authoritative than the human guess they replaced.

The citation is the dangerous part. An answer that points at a source reads as verified. If the source is a page nobody has updated in three years, the citation has laundered a stale fact into an apparently sourced one.

This is Module 1’s quiet-failure mode reached by a new route. Nothing errors. Accuracy metrics measured against the corpus look fine, because the system is faithfully reproducing what the corpus says. The corpus is wrong.

The audit you run before building anything

A day or two of work, no infrastructure required.

Take a sample of real requests that the projection claims are answerable — fifty is enough to be informative. For each one, try to answer it yourself using only the documentation, and record which of four things happened:

graph TD
  Q["Try to answer 50 real<br/>requests from the<br/>documentation alone"] --> A["<b>Answered cleanly</b><br/><i>retrieval will work</i>"]
  Q --> B["<b>Answer exists but<br/>is hard to find</b><br/><i>chunking and metadata</i>"]
  Q --> C["<b>Answer is stale or<br/>contradicted elsewhere</b><br/><i>documentation work —<br/>not an AI problem</i>"]
  Q --> D["<b>Answer does not exist</b><br/><i>content creation —<br/>not an engineering task</i>"]

  classDef start fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef win fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef fix fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef not fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class Q start
  class A win
  class B fix
  class C,D not

Four outcomes. Only the olive one is a retrieval success. The two terracotta ones are not solvable by anything in this module, and their share is the real finding.

Answered cleanly — retrieval will work here. This category’s share is your realistic deflection ceiling, and it is almost always lower than the headline projection.

Answer exists but is hard to find — a retrieval problem, and the one this module can fix. Better chunking, better metadata, structural boundaries. Genuinely addressable.

Answer is stale, or contradicted somewhere else — documentation work. No chunking strategy resolves two pages that disagree; the system will confidently pick one. You have just discovered that a content team is on your critical path and does not know it.

Answer does not exist — content creation. Not an engineering task, not in your remit, and not something a model supplies without inventing it.

Those last two categories are the finding. Their combined share tells you how much of the projected benefit is actually a documentation project wearing an AI costume.

The conversation this forces

If the audit shows a large share in the bottom two categories, you have to tell your sponsor that the headline number is optimistic. In month one, with evidence, before the commitment hardens.

This is uncomfortable and it is the job. Module 1 argued that an architect who finds the business case does not close has done the most valuable thing available in week five; this is that argument in concrete form. The alternative is shipping on schedule against a target that was never reachable, and being the person who did not notice.

It also reframes the proposal in a way that is often better received than expected: the initiative becomes two tracks — a retrieval system over the documentation that is good, and a documentation remediation effort for the part that is not, with a named owner and its own timeline. That is a harder plan to approve and a much easier plan to deliver.

The discipline that fixes it is not new

Documentation that retrieves well has the properties documentation has always needed: a single source of truth per fact, explicit scope on each page, structural headings, dates, and withdrawal of material that is no longer true.

This is Module 1’s Documentation as Code thesis, and the convenient part is that there is no trade-off between the two audiences. Structuring documentation so retrieval works is the same work as structuring it so a new engineer can use it. The investment serves both, which makes it considerably easier to justify than a project framed as “preparing our docs for AI”.

One honest caveat: exhaustive documentation is not the goal and can be worse than sparse documentation, because volume dilutes retrieval and multiplies the surface that goes stale. Fewer, better-maintained, clearly-scoped pages beat comprehensive coverage nobody updates.

Testing the 38%

The audit found that 38% of ~12,000 monthly tickets — about 4,560 — are “how do I…” questions the documentation already answers. Every number downstream rests on it: the $3.00 blended target, the $1.54 per-interaction ceiling, Maya Ortiz’s headcount relief, Priya Shah’s paid tier.

The test: sample 50 tickets from that 4,560, and have someone who knows the product try to answer each from the documentation alone. Grace Adeyemi’s team is the right source — they have seen the volume and know which questions are traps — and this is a real capacity ask, measured in a day or two, which Module 3’s displacement discipline says must be named rather than assumed free.

What each outcome would mean here. A high share answered cleanly validates the business case and the deflection target stands. A high share hard-to-find is good news for this module — it is exactly what chunking and metadata work addresses. A high share stale or contradictory means the initiative has a dependency on documentation ownership that nobody has scoped, and the 38% is not a deflection rate but an aspiration.

And the uncomfortable possibility. The original audit classified tickets as “answerable from existing documentation”. It is worth knowing how that judgement was made — whether anyone actually attempted the answers, or whether tickets were categorised by topic and assumed answerable. Those are very different claims, and only one of them supports a $1.2M commitment.

Asking that question in month one costs nothing and is exactly the kind of question a new architect should be asking. Asking it in month six, after the quarter has been committed, is a different conversation entirely.

Feeds the workbook

Your data and embedding strategy should state this assumption explicitly and say how you would test it. The rubric does not require you to have run the audit — it requires that the documentation-quality dependency is visible rather than silently assumed away.

Check your understanding

1. Design the 50-ticket audit for the case system. Who runs it, how long does it take, and what gets displaced to make room?
2. Your audit finds 40% of the sample stale or contradictory. Write the two sentences you would say to your sponsor.
3. Someone proposes indexing everything, on the grounds that more documentation means better retrieval. What is the counter-argument?

---

→ Twelve topics done. The stack is chosen and its hardest constraint is not negotiable. — Module 5 — Delegating to Agents

Optional workbook: Exercise 4 turns this module into artifacts. The exercises build on each other, so work them in order or not at all.