What Fine-Tuning Actually Costs

Overview

The previous topic put fine-tuning last and promised an explanation. This is it.

The training run is not the expensive part. The expensive parts are the dataset before it and the obligation after it — and the obligation is the one that does not appear in any proposal.

Four costs, in ascending order of surprise

The dataset. Labelled examples at sufficient quality and quantity, covering the behaviour you want including its edge cases. This is skilled work by people who understand the domain, and in Module 1’s terms it is Domain Authority being spent. It is almost always the largest line item, and it is almost always underestimated because it looks like data entry and is actually judgement.

The training run. Real money, bounded and predictable. Of the four, this is the one people plan for, and it is usually the smallest.

Evaluation per version. Every tuned variant needs its own run against your golden set. You are no longer evaluating a model; you are evaluating a model family, and the surface grows with each variant you keep.

The obligation: every base model upgrade invalidates your work. This is the cost that changes the decision.

The ratchet

A tuned model is pinned to the base version it was trained on. The frontier does not stop.

graph LR
  B1["Base v1"] --> B2["Base v2<br/><i>better, cheaper</i>"]
  B2 --> B3["Base v3<br/><i>better again</i>"]
  T1["Your tuned model<br/>pinned to v1"] -.->|"still on v1"| T2["Still on v1"]
  T2 -.->|"still on v1"| T3["Still on v1<br/><i>now behind on<br/>quality and price</i>"]

  classDef base fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef stuck fill:#D8D3CB,stroke:#B8B2A8,stroke-width:2px,color:#3F3F3F
  class B1,B2,B3 base
  class T1,T2,T3 stuck

The amber row advances without you. The drained row is what you own. Catching up means re-running the whole exercise — dataset, training, evaluation — on every upgrade you want.

So the real question is not “can we afford to fine-tune?” It is “can we afford to fine-tune repeatedly, on someone else’s release schedule, indefinitely?” If the honest answer is no, you are choosing to freeze at a version — which may be fine, and should at least be a decision rather than a discovery.

There is a second-order effect worth naming. Because the tuned artefact is non-portable, fine-tuning raises the cost of every lock-in type from the previous topic simultaneously. The switching cost you measured in weeks becomes a switching cost measured in a re-run of the whole programme.

When it is genuinely right

The technique is not a trap; it is a specialised tool with a narrow fit. Four situations where it earns its cost:

A narrow, stable task. Classification into a taxonomy that does not change. Extraction into a fixed schema. Behaviour that has been settled for years. The snapshot problem does not bite where nothing moves.

Behaviour prompting cannot reach reliably. Sometimes a format or style requirement survives every prompt formulation and still fails a few percent of the time. If that few percent matters and prompting has genuinely plateaued — measured, not felt — tuning is the remaining lever.

High volume where unit cost dominates. A small tuned model that matches a large prompted one on a narrow task can be dramatically cheaper per interaction. At sufficient volume, that arithmetic justifies the whole exercise. The threshold is a real number: work out where the dataset and retraining cost is repaid, and check that your volume clears it.

Latency floors a large model cannot meet. A smaller tuned model is faster. If your p95 budget is genuinely unachievable otherwise, this is a legitimate route.

What it does not fix, restated because it is where the money goes: factual freshness, missing context, and incapability at the task.

State the uncertainty

Two honest caveats.

The tooling is improving and costs are falling. The ordering in this topic — prompting, then retrieval, then tuning — reflects current economics rather than a permanent law. If dataset construction becomes cheap and retraining becomes routine, the calculus shifts, and the thing to watch is the maintenance cost rather than the training price.

And the plateau judgement is genuinely difficult. “Prompting has plateaued” is frequently claimed after a few attempts by someone who wanted to try tuning. Require evidence: a measured curve across several serious prompt formulations on your golden set, not an impression.

Rejected, with the condition that would change it

Fine-tuning is rejected for the support assistant, and the reasons go in the ADR:

• The knowledge changes weekly, so a snapshot is wrong by construction — the diagnostic in the previous topic already settled this.
• It would add the least portable artefact in the stack while Module 3’s bet 1 is explicitly keeping provider switching cheap, with a reversal condition that assumes it stays cheap.
• Nobody has demonstrated a prompting plateau. There is no measured curve, because there is not yet a system.
• There is no capacity for it. Dataset construction would consume Grace Adeyemi’s team — the only people who know which of 12,000 monthly tickets are genuinely hard — and Module 3’s displacement statement already spends the one engineering slot available.

Where it might later be right. A narrow ticket classifier: routing an incoming ticket to a category or to the right queue. That task is bounded, its taxonomy is comparatively stable, and it runs on every ticket rather than on the deflectable subset — so unit cost matters more and freshness matters less.

The reconsideration trigger, written in the bet format: revisit fine-tuning for the classifier if classification volume exceeds the point where a small tuned model repays its dataset cost within two quarters, and a measured prompting curve on a labelled sample shows a genuine plateau. Both conditions, not either.

Note what makes this a decision rather than a dismissal: it names the task, the threshold and the evidence that would overturn it. A rejection with a trigger survives being proposed again in six months by someone new.

Feeds the workbook

Your prompting / RAG / fine-tuning decision should record fine-tuning’s rejection with its reasons and its reconsideration trigger, in the bet format from Module 3. A decision with no trigger will be re-litigated.

Check your understanding

1. Someone proposes fine-tuning to “teach it our domain”. Which of the four costs would you put in front of them first, and why that one?
2. Work out the volume at which a tuned small model repays its dataset cost for a narrow task. Which assumption is doing the most work in your arithmetic?
3. How would you establish — with evidence rather than impression — that prompting has plateaued?

---

→ That closes Section 1: you know what does the thinking. Now, where does it run — and who is allowed to see the data on its way there? — Hosted or self-hosted