Embedding Pipelines and Refresh

Overview

An index is not built once. It is a product with a lifecycle, and the lifecycle contains two problems that are easy to defer and expensive to retrofit: keeping it current, and getting things out of it.

The second one is a one-way door. Once personal data has been embedded, removing it is substantially harder than never putting it there.

Refresh semantics

Three models, with honest costs:

Model	How it works	Cost
Full rebuild	Re-process everything on a schedule	Simple and predictable; pays the whole embedding bill every cycle, and freshness is bounded by cycle length
Incremental	Process only what changed	Cheap per cycle; requires reliable change detection and a deletion path, which is where the bugs live
Event-driven	Re-process on publish	Best freshness; couples your pipeline to another team’s system, which is a new dependency and a new failure mode

Module 3 made this choice once already, as a tactic: invalidate on publish rather than shortening a staleness window. The same logic applies to the index, with the same cost — the documentation system now has to emit an event your pipeline consumes, which means a conversation with whoever owns it.

The hard part is deletion, not insertion

Insertions are easy. Everything else is where systems go wrong.

A document is updated and the old vectors remain: retrieval now returns both versions, the model sees contradictory source material, and the answer is confidently wrong in a way no monitoring catches. A document is withdrawn and its vectors remain: the system cites a policy that no longer exists. A document is moved or renamed and change detection treats it as new: you have duplicates competing in retrieval.

So the pipeline requires, as explicit design elements:

A stable identity per source document that survives renames.
A mapping from every vector back to its source, so that removing a document removes exactly its vectors.
Update-as-replace semantics, so a revision cannot leave its predecessor behind.
A tombstone or verification step confirming that deletions actually removed what they claimed to.

None of this is difficult. All of it is routinely absent, because insertions are what gets tested during development and deletions are what happens in production.

Personal data before embedding

Now the one-way door.

Embeddings are derived data. A vector produced from text containing someone’s name and address is not a copy of that text, but it was derived from it, and data-protection obligations — the right to erasure in particular — reach derived data. A right-to-erasure request that you satisfy in the primary store while leaving the derived vectors in place has not been satisfied.

Vectors are also awkward to inspect. You cannot grep an index for a phone number. Which means the practical answer is not to find personal data later but to prevent it entering:

graph LR
  S["Source<br/>document"] --> R["<b>Redaction</b><br/>detect and mask<br/>personal data"]
  R --> C["Chunk<br/>+ metadata"]
  C --> E["Embed"]
  E --> I["Index"]
  I -.->|"erasure request:<br/>vector → source<br/>mapping"| S

  classDef plain fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef gate fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  classDef store fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  class S,C,E plain
  class R gate
  class I store

The redaction stage sits before embedding, not after indexing. The dotted return path is the erasure route, and it only exists if the vector-to-source mapping was built in.

Four requirements follow, and the workbook rubric asks for the first one specifically:

Redaction before embedding. Detection and masking of names, addresses, phone numbers and payment references as a pipeline stage, upstream of the embedding call.

A vector-to-source mapping. Without it, erasure is impossible and you will rebuild the index to satisfy a single request.

A documented erasure procedure. Not code — a procedure your compliance owner can read, covering primary store, index, caches and logs. Erasure that stops at the database is incomplete, and caches are the component people forget.

Evidence that it works. Sampling redaction output, counting what was masked, and retaining enough to show an auditor.

What redaction costs

Stating it plainly, because redaction is usually presented as free.

It removes signal. A support conversation with every name, address and reference stripped may lose the thread that made it useful — “the technician at the Oak Street site couldn’t access the work order” becomes considerably less informative. Over-redaction produces a corpus that is compliant and unhelpful.

Detection is imperfect in both directions. It misses things, and it masks things that were not personal data. Both failures have costs, and the acceptable rates are a decision for your compliance owner and the team using the tool — Module 2’s refusal-rate pattern again.

It is irreversible by design. Which is the point, and means a redaction bug discovered later is a re-processing project.

Given all that, the honest question for any corpus containing personal data is not only how to redact it, but whether to index it at all. A corpus whose value survives redaction is worth indexing; one whose value depends on the personal details may not be.

The ticket corpus is the hard half

Documentation is straightforward: structured, no personal data, event-driven refresh on publish to satisfy the Module 2 scenario about answering from documentation updated within the last seven days.

Historical tickets are the problem. They contain names, site addresses, phone numbers and occasionally payment references, across years of accumulated history at roughly 12,000 a month.

The pipeline: redaction stage before chunking and embedding; per-ticket stable identity; vector-to-source mapping retained; update-as-replace; a verified deletion path.

The erasure path: a customer exercising their right to erasure triggers removal from the primary store, from the index via the vector mapping, from the semantic cache if one holds generated answers derived from their data, and from traces retained for observability. All four, with the procedure written where Lena Fischer can read it — she has blocking authority and a breach with her name on it is what she fears, so a procedure she has reviewed is worth more than an architecture she has not.

The open question worth raising early: does the ticket corpus still carry value after redaction? The deflection case rests on the 38% of questions the documentation answers. If redacted tickets add little beyond what documentation already covers, the cheapest compliant design may be to index documentation only and treat historical tickets as a later, separately-justified increment.

That would remove an entire category of compliance risk, shrink the pipeline, and cost one source of marginal retrieval quality. Whether it is right depends on evidence nobody has yet — which makes it a bet with a falsification condition, not a decision to make by assumption.

Feeds the workbook

This is the core of your data and embedding strategy. Two rubric items land here: personal data addressed at the pipeline stage before embeddings exist, and refresh strategy specified rather than assumed. Include the erasure path — Workbook exercise 6’s rubric asks about erasure from embeddings specifically.

→ The pipeline is sound and the data is clean. One assumption remains unexamined, and the whole business case rests on it. — The documentation quality problem