Prompting, RAG or Fine-Tuning

Overview

Three techniques, routinely treated as a progression from basic to advanced. They are not a progression. They are answers to three different problems, and applying one to the wrong problem is how teams spend a quarter achieving nothing.

The framework is a single diagnostic question: what is actually broken?

The diagnostic

graph TD
  Q{"What is<br/>actually wrong?"} -->|"doesn't know<br/>your facts"| R["<b>Retrieval</b><br/>supply the facts<br/>at request time"]
  Q -->|"knows, but<br/>behaves wrong"| P["<b>Prompting</b><br/>format, tone, refusals,<br/>escalation rules"]
  Q -->|"behaviour won't hold<br/>after prompting plateaus"| F["<b>Fine-tuning</b><br/>narrow, stable tasks only"]
  Q -->|"can't do the<br/>task at all"| M["<b>A different model,<br/>or a smaller task</b><br/><i>neither RAG nor tuning<br/>fixes incapability</i>"]

  classDef gate fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef first fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef mid fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef last fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class Q gate
  class P,R first
  class F mid
  class M last

One question, four answers. The terracotta leaf is the one teams refuse to accept, and it is where most wasted quarters go.

It doesn’t know your facts → retrieval. Your product documentation, your policies, your historical records. No amount of prompt engineering conjures information the model never saw, and training it in is a slow, expensive way to achieve what supplying it at request time achieves immediately.

It knows, but behaves wrong → prompting. Wrong format, wrong tone, answers when it should refuse, continues when it should escalate. These are instruction problems. They are also, usefully, the cheapest problems to iterate on: the feedback loop is minutes.

Prompting has plateaued on a narrow, stable behaviour → consider fine-tuning. Not before. The next topic is about what that actually costs.

It can’t do the task at all → a different model, or a smaller task. This is the answer nobody wants, and refusing it is the most expensive mistake in the list. Neither retrieval nor fine-tuning fixes incapability. If a model cannot reliably perform the reasoning the task requires, supplying more context and training on more examples produce a confident, fluent, wrong system faster.

The freshness test settles most cases

A second question, and often decisive on its own: how often does the knowledge change?

Knowledge that changes weekly cannot live in trained weights. Fine-tuning bakes in a snapshot; the moment your documentation is revised, the model is confidently describing a product that no longer exists, and the only remedy is retraining. Retrieval reads current material at request time, which is why it is the right answer for anything that moves.

Conversely, behaviour that has been stable for years — a fixed output schema, a consistent house style, a classification taxonomy that does not change — is a legitimate candidate for training, because the snapshot problem does not apply.

So: changing facts take retrieval; stable behaviour may take tuning. That single line resolves the majority of real cases.

Order of attempts

Try them in increasing order of cost and decreasing order of reversibility:

1. Prompting. Minutes to iterate, free to abandon.
2. Retrieval. Days to weeks, and it brings the corpus-quality dependency below.
3. Fine-tuning. Weeks, with a recurring maintenance cost and the least portable artefact in your stack.

Hybrids are normal and not a compromise. Retrieval for the facts plus careful prompting for refusal and escalation behaviour is the common shape; retrieval plus a small tuned model for a narrow formatting or classification step inside the flow is also legitimate.

The “just add RAG” reflex

Retrieval has become the default answer to every AI problem, and it deserves a specific warning.

Retrieval amplifies your corpus quality in both directions. Over well-maintained documentation, it produces grounded, current, checkable answers. Over documentation that is stale, contradictory, or wrong, it produces confident answers that are stale, contradictory or wrong — and it produces them faster and at greater scale than a human ever did, with a citation attached that makes them look more trustworthy.

This is why the last topic of this module is about documentation quality rather than about infrastructure. A retrieval system is a quality multiplier with no sign attached; you supply the sign. Module 7 returns to this as an antipattern with a detection signal.

The other cost of retrieval, understated: it is the layer with the highest reversal cost in the stack, because chunking and embedding decisions are baked into an index that must be rebuilt to change them.

The diagnosis

Run the diagnostic against the support platform.

What is wrong? An internal audit found 38% of ~12,000 monthly tickets are “how do I…” questions the documentation already answers. The model does not know Lintel’s product. That is squarely the first branch: it doesn’t know your facts.

How often does the knowledge change? The product ships continuously and documentation is revised accordingly — Module 2’s accuracy scenario explicitly covers answering from documentation updated within the last seven days. Weekly-changing knowledge rules out trained weights.

Decision: retrieval over the docs corpus and historical tickets, with prompting for behaviour. The refusal and escalation boundaries from the Module 2 intent statement — never discuss contracts or pricing, escalate on contradictory documentation, escalate any credit request — are instruction problems, handled in the prompt and enforced at the serving layer.

Fine-tuning: rejected, with reasons recorded. The facts move weekly, so a snapshot is wrong by construction. It would add the least portable artefact in the stack at exactly the moment Module 3’s bet 1 is keeping provider switching cheap. And it does not address anything that is actually broken.

One honest caveat. The 38% figure assumes the documentation answers those questions well. The diagnostic says retrieval; it does not promise the corpus can support it. If the documentation turns out to be thin or contradictory, retrieval will deflect confidently and wrongly, which is the 2023 failure mode arriving by a new route. That assumption gets tested in this module’s final topic, not assumed.

Feeds the workbook

This is the prompting / RAG / fine-tuning decision artifact. The rubric asks that it state what it is optimising for and what it gives up — so record the rejected options with their reasons, not just the choice.

Check your understanding

1. Run the diagnostic on a problem from your own work. Which branch did it land on, and which technique had you assumed?
2. A stakeholder insists fine-tuning will “make it understand our domain”. What question do you ask to locate the actual fault?
3. Your corpus is of unknown quality and retrieval is the indicated answer. What do you do before building anything?

---

→ Fine-tuning was deferred twice now, with a promise to explain. Here is the bill. — What fine-tuning actually costs