AI Applied to Legacy

Overview

Capacity is the binding constraint, so anything that multiplies it deserves a careful look. Legacy work is also unusually well suited to assistance: the task is mostly comprehension, the context is all in the repository, and the output is verifiable.

But one of the obvious applications has a failure mode that defeats its entire purpose, and it is not obvious until it has already happened.

Where it genuinely helps

Explaining unfamiliar code. The strongest use. A model reading a long method and producing a summary of what it appears to do is a substantial accelerator for the slowest part of legacy work, and it is cheap to verify: you are going to read the code anyway, and now you read it with a hypothesis.

Critically, this attacks the exact bottleneck the previous topic identified — the need for someone with deep context. It does not replace that person, but it raises how much a less experienced engineer can do before needing them, which is the constraint that matters.

Finding call sites and tracing data flow. Mechanical, tedious, and the kind of exhaustive sweep where a model’s patience beats a human’s. Verification is direct: the references either exist or they do not.

Proposing refactoring steps. Given a target structure, enumerating the sequence of safe intermediate moves is work a model does well, because the moves are standard and the hard part is remembering them all.

Drafting documentation of current behaviour — with the Module 9 discipline attached. The output is a hypothesis to verify and must be marked as reconstructed, never filed as a contemporaneous record.

The trap: characterization tests

The application that looks most valuable and carries a specific, severe failure mode.

Recall the purpose from the earlier topic. A characterization test captures what the system actually does, including behaviour nobody intended, so that you can change the code and detect any difference. Its whole value lies in being an honest record of present behaviour, bugs included. It is not a specification.

Now consider what a model does when asked to write tests for a function. It infers intent — from the function’s name, its comments, the surrounding conventions, and everything it has learned about what code like this is supposed to do. It writes tests asserting the intended behaviour.

That is the correct instinct for new code and exactly wrong here:

A characterization test that asserts intended behaviour will fail against the current system, and the natural response is to “fix” the discrepancy — which is a behaviour change, made unknowingly, with no test protecting you.

Worse, the discrepancies are precisely the interesting cases: the undocumented edge case, the workaround for one large customer, the quiet fix for a recurring problem. The generated test disagrees with reality exactly where reality holds something load-bearing that nobody remembers.

graph TD
  A["Ask for tests on<br/>legacy code"] --> B["Model infers<br/><b>intended</b> behaviour"]
  B --> C["Tests fail against<br/>the real system"]
  C --> D["<b>The discrepancy is the<br/>behaviour that matters</b><br/><i>and it looks like a bug</i>"]

  classDef start fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef mid fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef bad fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class A start
  class B,C mid
  class D bad

The failure is not that the tests are wrong. It is that they are wrong in the one direction that destroys their purpose.

The fix is procedural and simple. Generate characterization tests by running the code and recording what it returns, then use the model to write the test around the observed values rather than to predict them. The model is excellent at enumerating the input cases to cover and at writing the scaffolding; it must not be the source of the expected values. Any case where the generated expectation and the observed behaviour disagree is a finding to investigate, never a discrepancy to correct.

The general limit

Module 7’s point, which holds throughout: fluency is uncorrelated with correctness, and legacy code is the worst possible place for a confident guess because nobody remembers enough to notice.

The pattern that makes this safe is consistent across every use above: a model is reliable here when its output is cheap to verify against something real — the code, the references, the recorded behaviour. It is unreliable whenever it is asked to supply a fact that exists only in a history nobody wrote down. Comprehension, yes. Archaeology, no.

Where it accelerates, and where it was nearly used wrong

Applied to the stage-two work from the previous topic — characterization tests on the monolith’s credit and scheduling behaviour.

Where it earns its place. The 2016 Rails code handling credits has accumulated nine years of conditionals. A model reading it produces a summary of the apparent logic in minutes, which is a genuine multiplier on the one resource the sequence is constrained by: it lets the new engineers arrive at Ivan Petrov’s door with specific questions rather than general confusion. The review is still his; the preparation is no longer.

It is also the right tool for enumerating every call site of the credit path across the monolith and the twenty Go services — exhaustive, mechanical, and directly verifiable.

Where the trap nearly bit. Asked to write tests for the credit logic, the generated tests asserted what the function obviously ought to do: validate the amount, check the customer’s eligibility, apply the credit, record an audit entry.

Run against the real system, several failed. The reason they failed is the thing that matters: the collaborative design session in Module 9 found that credits are two processes under one name — a goodwill credit any agent can issue, and a contractual adjustment requiring finance approval. The live code handles both through one path with a branch nobody documented, and the generated tests had smoothed that into a single coherent intended behaviour.

Had those tests been accepted and the “failures” fixed, the contractual-adjustment path would have been quietly altered — in the exact area where Module 5’s action tiering was already wrong, and where an error is a customer-visible financial one.

So the procedure stands. Record the actual returns for a set of real inputs; have the model write the tests around those recorded values; treat every disagreement between its expectation and the observation as a finding. That finding, in this case, was worth more than the tests: it independently confirmed the two-tier split the design session discovered, which is the second piece of evidence for a correction to the action list.

Feeds the workbook

This shapes the execution of your migration sequence: where assistance multiplies scarce context, and the explicit rule that generated expectations are never the source of truth for current behaviour.

→ One more decision, and the frontier moves under it every quarter. — Buy vs build under a moving frontier