Retry Semantics

Overview

Retrying a failed call is so routine it barely registers as a decision. In a system that takes real actions it is one of the most consequential decisions in the design.

Retry mechanics are prerequisite material. What this topic adds is the specific shape the problem takes when a non-deterministic component decides what to retry, and the state that most designs have no answer for.

Three outcomes, not two

A call does not succeed or fail. It resolves into one of three states, and the third is where the damage lives.

graph TD
  A["Action attempted"] --> B["<b>Succeeded</b><br/>confirmed by response"]
  A --> C["<b>Failed</b><br/>confirmed not applied"]
  A --> D["<b>Unknown</b><br/>timeout, connection lost,<br/>ambiguous error"]
  B --> E["Do not retry"]
  C --> F["Safe to retry"]
  D --> G["<b>Verify, then decide</b><br/>never retry blind"]

  classDef ok fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef fail fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef unknown fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  classDef act fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  class B,E ok
  class C,F fail
  class D unknown
  class A,G act

Three outcomes. Most retry logic handles two of them, and the terracotta branch — where you genuinely do not know — is the one that produces duplicate credits.

The unknown state arises constantly: a timeout where the request may have been processed after you stopped waiting, a dropped connection after the write committed, an error response from a proxy rather than from the service. In a system whose actions move money, treating unknown as failed is how a customer receives two credits.

The classification every action needs

Three categories, and every tool on your surface belongs in exactly one:

Safe to retry freely. Reads and idempotent lookups. No state changes, so a duplicate costs only tokens and latency.

Safe to retry with a key. Writes carrying an idempotency key, where the downstream system recognises a repeat and returns the original result rather than applying the change twice. This is the category most consequential actions should be engineered into.

Never retry blind. Anything with an external side effect that cannot be deduplicated — a message sent, a third-party call, an irreversible change. Here the only safe path is verification.

The key must be issued by your code and tied to the logical intent, not generated by the model. Module 5 made this point and it is worth repeating because the failure is silent: a model asked to supply an idempotency key may supply a fresh one on the retry, which defeats the entire mechanism while looking correct in every log.

Verify, then decide

For the unknown state, the pattern is not retry — it is read before write.

Query the downstream system for the effect: does a credit against this ticket with this key already exist? Was the job already moved to this slot? That turns an unknown into a known, and the retry decision becomes trivial.

Two requirements fall out, and both are architectural:

• Every consequential action needs a corresponding read that can establish whether it happened. If a tool can do something but nothing can check it, the unknown state has no resolution and you are left guessing.
• The verification must be reliable when the write path is not. If both go through the same failing dependency, verification is unavailable exactly when you need it. The honest answer then is to stop and escalate to a human rather than to guess — which is a degradation-ladder rung, the subject of the next topic.

Where non-determinism makes this worse

Two complications that do not exist in a conventional system.

The model may retry on its own initiative. Inside a loop, a tool call that returns an error is an observation, and the system may simply try again — possibly with slightly different arguments, which defeats key-based deduplication because the key is derived from the arguments. The bound from Module 5 is what contains this, and the enforcement must be in the action layer: this intent has already been executed is a check your code makes, not a rule the model follows.

Retries interact with cost in a way Module 6 quantified. A retry inside a loop resends the whole accumulated context, so retry storms are expensive as well as dangerous. A retry policy with no cap is a cost incident waiting for a bad afternoon.

What a retry policy costs

Stating it, because “just add retries” sounds free.

Idempotency keys require downstream support — if the monolith’s credit endpoint does not honour them, you cannot have category two, and someone has to build that. Verification reads add latency to the failure path, which is the path already under time pressure. And a conservative policy means some genuinely transient failures become handoffs, which costs deflection.

That last trade is the right one. A handoff costs a few dollars of human time; a duplicate credit costs money and trust, and arrives in a report with your name near it.

The policy, action by action

Action	Category	Key	Unknown-state behaviour
search_docs, read_job, check_entitlement	Retry freely	—	Retry with backoff, capped
reschedule_job	Retry with key	job id + target slot + request id	Read the job; if already at the target slot, treat as succeeded
issue_credit	Retry with key, or not at all	ticket id + amount + request id	Read first. If a credit with this key exists, stop and report success. If the read is unavailable, hand off
reset_device	Never retry blind	device id + request id	Read device state; if already reset within the window, treat as succeeded

The dependency this exposes. Category two requires the Rails monolith’s endpoints to honour idempotency keys. If they do not — and a 2016 monolith may well not — then issue_credit cannot be retried at all, and every unknown outcome becomes a handoff to one of the 45 agents with the ambiguity stated plainly.

That is a question for Ivan Petrov, who owns the monolith, and it is better asked in week one than discovered during an incident. It also changes the deflection arithmetic: if unknown outcomes become handoffs, the handoff rate rises by however often the integration is flaky, which Module 2’s 15% ceiling has to absorb.

The cap. Three attempts on reads, one verification-then-decide cycle on writes, and no retry at all once the step cap of 12 is reached — because at that point the loop is already at its bound and a retry would exceed it.

What the human receives when an unknown outcome escalates: the action attempted, its key, the verification result or the reason verification failed, and an explicit statement that it is not known whether this was applied. Module 5’s handoff package has a field for exactly this, and this is the case it was designed for.

Feeds the workbook

The rubric requires your uncertainty policy to account for actions with side effects — a retried credit being a second credit is the named example. Build the table, classify every action, and write the unknown-state behaviour explicitly. That column is the one that distinguishes a policy from a hope.

Check your understanding

1. Classify every action in your design into the three categories. Which ones lack a corresponding read?
2. The downstream system does not support idempotency keys. What changes in your policy, and which Module 2 number moves?
3. The model retries a tool call with slightly different arguments. Why does key-based deduplication fail, and where do you fix it?

---

→ A retry policy decides what to try again. The next question is what to do when nothing will work. — Fallback and degradation