Layers of the AI Stack

Overview

Before choosing anything, a map. An AI system decomposes into five layers, each owning a different kind of decision.

The useful property of the map is not the layer names — it is that the layers have wildly different reversal costs. Two of them you can change in a sprint. One of them, changed badly, means reprocessing your entire corpus.

This module stays at the altitude where an architect decides. For the engineering underneath it, Chip Huyen’s AI Engineering and Designing Machine Learning Systems are the reference works, and the pricing and latency mechanics quoted throughout come from the providers’ own production documentation — which is where to check them, because they change.

The five layers

graph TD
  A["<b>Foundation model</b><br/>which model answers<br/><i>reversible if you kept a seam</i>"] --> B["<b>Orchestration</b><br/>how calls are sequenced<br/><i>reversible, often unnecessary</i>"]
  B --> C["<b>Retrieval infrastructure</b><br/>index, store, chunking, embeddings<br/><i>expensive — changing it reprocesses everything</i>"]
  C --> D["<b>Serving layer</b><br/>gateway, cache, rate limits, routing<br/><i>moderate — but it is where policy lives</i>"]
  D --> E["<b>Observability</b><br/>traces, evals, cost attribution<br/><i>cheap to add, painful to add late</i>"]

  classDef cheap fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef mid fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef dear fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class A,B cheap
  class D,E mid
  class C dear

Five layers, coloured by what it costs to change your mind. The terracotta one is the layer most teams decide by default and never revisit.

Foundation model. Which model answers, at what reasoning effort. The decision feels momentous and is among the cheaper ones to revisit — provided you kept a seam at the invocation boundary. Without one, it is a refactor across every call site.

Orchestration. How calls are sequenced: one call, a chain, a retrieval-then-generate flow, or an agent loop. The honest default here is the leftmost thing that works, which Module 1 argued and Module 5 will argue at length. Many systems need no orchestration framework at all, and adopting one early buys a dependency in exchange for abstractions you have not yet needed.

Retrieval infrastructure. The index, the store, and — critically — the chunking and embedding decisions baked into it. This is the expensive layer, and the reason is mechanical: changing how documents are split or which embedding model represents them invalidates every vector you hold. You do not edit this decision; you re-process the corpus and re-index it.

Serving layer. The gateway in front of the model: caching, rate limiting, routing, timeouts, retries. Structurally unremarkable and strategically important, because it is the one place cross-cutting policy can live. Guardrails belong here, as do the cost controls from Module 2 and the degradation ladder from Module 3.

Observability. Traces, evaluation runs, cost attribution per interaction. Cheap to build in and genuinely painful to retrofit, because the thing you most need — the record of why a specific past interaction went wrong — cannot be reconstructed after the fact. Module 1’s quiet-failure problem is an observability problem wearing a different hat.

The layers interact, which is the part that surprises people

The map implies independence. The reality is a chain where one choice propagates:

A model’s context window bounds how much retrieved material you can send. That bound sets a practical chunk size and a retrieval depth. Chunk size changes retrieval precision — small chunks match tightly but carry little context; large chunks carry context but dilute the match. Retrieval depth multiplies into input tokens, which is most of your cost per interaction. And cost per interaction is the number you promised the person holding the budget.

So “which model?” is not a question you can answer and move on from. It constrains three layers below it, and a change there reaches all the way down to the unit economics in your NFR budget.

The practical consequence: decide the layers in dependency order, not in order of interest. Most teams start at the model because it is the most interesting layer, then discover their chunking strategy was implied by a decision they made in week one.

Fewer layers is a real option

Each layer added costs latency, operational burden, and another thing that can fail. The instinct to assemble all five from the start should be resisted.

A genuinely useful question at each layer: what breaks if we leave this out for now? Omit orchestration and you write the sequencing yourself in ordinary code, which for a two-step flow is simpler than a framework. Omit a vector store and you may find that keyword search over a well-maintained corpus answers most questions — an unfashionable finding that occasionally survives measurement.

What you should not omit is observability, and that is the asymmetry worth internalising. It is the only layer whose absence you discover at the worst possible moment.

Which layer is the trap

Mapping the five layers onto the support platform:

Layer	The decision	Reversal cost
Foundation model	Which model, at what effort, in which region	Moderate — the reference architecture keeps the provider behind an interface
Orchestration	Retrieve-then-generate for answers; a bounded loop for the action path	Low — it is ordinary code in two services
Retrieval	How the docs corpus and historical tickets are split, embedded and indexed	High — re-chunking means re-embedding and re-indexing everything
Serving	Gateway holding the cache, the routing policy, the guardrails, the degradation ladder	Moderate, and it is where Lena Fischer’s controls are enforced
Observability	Per-interaction cost attribution, eval runs against the 300-question golden set, trace retention	Low to add now; impossible to backfill

The trap is the retrieval row, and it is the row nobody argues about in a kickoff. The corpus is product documentation plus historical tickets at ~12,000 a month — a body of text that grows continuously and contains personal data that must be handled before it is embedded. A chunking decision made casually in week two is a decision about how much every future re-index costs, and about whether the 38% deflection target is reachable at all.

Note which row the compliance constraints land on: the serving layer enforces them, but the retrieval layer is where the personal data physically ends up. Both need to satisfy Lena, and only one of them is cheap to fix later.

Feeds the workbook

Your AI stack selection needs a criteria matrix per layer. Before you build it, write the reversal cost next to each layer — it tells you where to spend your analysis, and the rubric’s requirement to name a decision that would be expensive to reverse is answered here rather than guessed at later.

Check your understanding

1. For a system you have worked on, which layer was decided by default rather than deliberately? What did it cost?
2. Trace the chain in this topic in the opposite direction: your cost per interaction is twice the target. Which layer’s decision do you examine first, and why not the model?
3. Which layer could you genuinely omit from the case system for a first release, and what breaks?

---

→ The map puts the model at the top. Choosing one is where most of the money is wasted, and benchmarks are why. — Choosing a model