Chunking as an Architectural Decision

Overview

Topic 1 identified retrieval infrastructure as the layer with the highest reversal cost. This is the decision inside it that does the damage — and it is typically made in an afternoon by whoever wrote the ingestion script.

How documents are split determines retrieval precision, which determines accuracy. It also determines how many tokens go into every request, which determines cost. One decision, both of the numbers you promised.

Why it is expensive to reverse

Chunk boundaries are baked into the index. Changing them means re-splitting the corpus, re-embedding every piece, and rebuilding the index — paying the embedding bill again and waiting for the pipeline to run.

For a small corpus that is an inconvenience. For a corpus of product documentation plus years of accumulated support history, it is a project with a budget. And since you cannot know the right chunking strategy before measuring retrieval quality, you will want to change it at least once.

Which produces the actual architectural requirement of this topic, and it is not a chunk size:

Build the indexing pipeline to be cheaply re-runnable from day one. Idempotent, resumable, versioned, and able to build a new index alongside the live one.

A team that can re-index in an afternoon will tune chunking until retrieval is good. A team for whom re-indexing is a two-week manual ordeal will ship whatever the first script produced and live with it for years. The difference is not skill; it is whether somebody made re-runnability a requirement before the first ingest.

The core trade-off

graph LR
  S["<b>Small chunks</b><br/>precise match,<br/>missing context<br/><i>answer is true<br/>but unusable</i>"] --> M["<b>Retrieve small,<br/>supply large</b><br/>match on the fragment,<br/>send the section"]
  M --> L["<b>Large chunks</b><br/>full context,<br/>diluted match<br/><i>retrieval misses,<br/>input bill grows</i>"]

  classDef bad fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  classDef good fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  class S,L bad
  class M good

Both ends fail, differently. The middle move costs more input tokens than small chunks and fewer than large ones — which is a trade, not a free lunch.

Small chunks match precisely and arrive without context. The retrieved fragment may be entirely true and useless — a step from a procedure with no indication of which procedure, or a condition with no statement of what it applies to. Worse, an answer that spans two chunks is retrievable only in pieces.

Large chunks carry their context and dilute the match: a chunk about many things is strongly similar to nothing in particular, so retrieval quality drops. They also inflate input tokens on every request, and from the previous topic, input volume is most of your bill.

Retrieve small, supply large is the move worth knowing: index fine-grained pieces for matching, then expand the hit to its surrounding section before sending it to the model. You get precise matching with usable context. The cost is more input tokens than small chunks alone — a real cost, paid on every request, against a ceiling you promised.

The dimensions to decide

Treat it as a criteria catalog rather than a single number:

• Boundary rule. Split on document structure — headings, sections, procedure steps — rather than at a fixed character count. Structural boundaries respect meaning; fixed-length ones cut sentences in half. This is the highest-leverage choice and it depends entirely on whether your documents have structure.
• Size and overlap. Overlap reduces the chance that an answer falls across a boundary, and costs storage and some duplicate retrieval.
• What is indexed versus what is sent. The retrieve-small-supply-large decision above.
• Metadata carried with each chunk. The one that matters most, below.

Metadata is where compliance lives — and it cannot be added later

Every chunk should carry, at index time: its source document, its version or effective date, its section path, the tenant or customer it belongs to if any, and its trust label from Module 2.

That list is not administrative tidiness. Each item enables a requirement you already wrote down:

• Tenant makes cross-customer isolation enforceable at the query, rather than hoped for in a prompt.
• Trust label is what lets the serving layer refuse to treat retrieved ticket text as instruction — the indirect-injection requirement from Module 2.
• Version and effective date are what let you detect staleness and answer “was this current when we said it?”
• Source is what makes a citation possible, and citation is most of what makes an answer checkable.

And here is the sting: you cannot add these afterwards without re-indexing. A corpus indexed without tenant metadata cannot have tenancy enforced at the query later; you rebuild it. This is the clearest example in the module of a cheap decision now versus an expensive one in six months.

Two corpora, two strategies

The sources are product documentation and historical tickets, and treating them as one corpus is the first mistake available.

Product documentation has structure — headings, procedures, feature sections. It chunks on structural boundaries, with each chunk carrying its section path, document version and effective date. Trust label: reviewed. Retrieve small, supply the enclosing section.

Historical tickets are conversational, unstructured, of wildly varying quality, and contain names, site addresses, phone numbers and occasional payment references. They chunk by conversation turn or resolution block, and they carry a customer tenant label and a trust label of untrusted — because anyone who ever wrote into support wrote that text, which makes it an indirect injection surface.

Three consequences fall out immediately:

1. The two corpora probably want separate indexes, or at minimum a hard metadata partition, because their trust levels and retention rules differ and a single mixed index makes both harder to reason about.
2. The personal data in tickets must be handled before embedding, not after — the subject of the next topic, and a one-way door.
3. Retrieval must be able to prefer documentation over tickets, since the deflection case is built on the 38% of questions the documentation already answers. A ticket that happens to match more strongly is not a better source.

The chunking decision for the ticket corpus is therefore not primarily about size. It is about what travels with each piece — and getting that wrong means re-processing years of support history to fix it.

Feeds the workbook

The rubric requires that chunking and refresh strategy be specified rather than left as “ingest the docs”. Specify the boundary rule, the metadata set, and the re-runnability requirement for the pipeline — that last one is the decision your future self will thank you for.

Check your understanding

1. Write the metadata set for the case system’s two corpora. Which field would be most expensive to add in six months?
2. Your documentation has no headings — a wall of text per page. What does that do to your boundary rule, and what does it imply about the work ahead?
3. Retrieve-small-supply-large raises input tokens on every request. How would you decide whether it is worth it, and against which number?

---

→ The index is designed. Now it has to stay current, and personal data has to be dealt with before it becomes a vector. — Embedding pipelines and refresh