Criteria Catalogs and Scale Calibration

Overview

The process says generate options and evaluate against criteria. That raises the question of what an architect actually needs to know in order to do either.

The answer is not a memorised catalog of technologies. It is a catalog of criteria — and a habit of checking what scale you are really operating at before applying any of them.

Carry criteria, not recommendations

A recommendation expires. The best queue, store or model for a job changes on a timescale of months, and an architect whose value rests on knowing today’s answer is depreciating in exactly the way Module 1 described.

A criteria catalog is the durable form of the same knowledge: for a class of decision, the dimensions that actually discriminate between the options, and the thresholds at which the answer changes.

The selection criteria for databases, queues, caches and load balancers are prerequisite material for this guide and are not re-taught. What is the architect’s job is having them ready, written down, and weighted for this organisation — so that an option can be evaluated in an afternoon rather than in a three-week spike.

Three properties make a criteria catalog worth maintaining:

It outlives the options. New entrants get evaluated against the same dimensions; the catalog does not need rewriting because a vendor shipped something.
It is auditable. A CFO cannot assess “we chose the better model”, but can absolutely assess a table of dimensions with evidence in each cell. That auditability is the whole reason the Module 8 justification pack works.
It exposes the dimensions nobody raised. Residency, exit cost, operational burden, and who can support it at 03:00 are criteria that get omitted precisely because they are nobody’s favourite topic.

The cost: building one takes real time, and a catalog left unmaintained becomes a confident record of last year’s thresholds — the same decay problem as the reference architecture, with the same fix of dating it and recording exceptions.

The radar, and why it is not yours yet

The technology radar — ThoughtWorks’ format, now widely copied — is the organisational companion to a private criteria catalog: technologies sorted into adopt, trial, assess and hold, published so many teams can see the portfolio position without asking.

It solves a real problem, and it is the wrong artifact for you today. A radar is a statement of position to an audience, which means it needs enough teams that asking individually does not scale, and enough mandate that the statement is credible. With neither, a radar reads as an edict from someone with no authority to issue one — and Module 8 is about why that costs more than it returns. The criteria catalog is the same knowledge in the form that requires nobody’s permission; Module 9 returns to the radar as something to introduce deliberately, later.

The discriminating-dimension test

A criterion earns its place only if options actually differ on it. Dimensions where everything scores the same are padding, and padding is how a comparison table becomes impressive and useless.

So for each candidate criterion, ask: would any two serious options score differently here? If not, cut it. A table with six discriminating criteria beats one with twenty where fourteen are ties — and it is also much harder to game, because there is nowhere for a preferred option to accumulate soft points.

Scale calibration

The second half of the topic, and the one that most often changes a design.

Every architectural answer is valid inside a range of numbers and invalid outside it. The discipline is to ask, before applying any criterion: at what number does this answer change?

graph LR
  A["~1 per minute<br/><i>one process,<br/>anything works</i>"] --> B["~10 per second<br/><i>ordinary service,<br/>cache and scale out</i>"]
  B --> C["~1,000 per second<br/><i>partitioning becomes<br/>structural</i>"]
  C --> D["~100,000 per second<br/><i>specialised design,<br/>different problem</i>"]

  classDef low fill:#7D9A4E,stroke:#5F7A38,stroke-width:2px,color:#ffffff
  classDef mid fill:#5B8FB9,stroke:#4A7A9E,stroke-width:2px,color:#ffffff
  classDef high fill:#E8A33D,stroke:#C4832A,stroke-width:2px,color:#ffffff
  classDef extreme fill:#D9695B,stroke:#B4483A,stroke-width:2px,color:#ffffff
  class A low
  class B mid
  class C high
  class D extreme

Order-of-magnitude bands, not a continuum. The design changes at the boundaries, and an answer validated in one band tells you very little about the next — the figures are rough and depend heavily on what each request does.

Two failure modes, and the second is more common in practice than the first.

Designing for a scale you will never reach. You pay the cost of partitioning, elaborate caching and elastic infrastructure now, and collect the benefit never. This is the expensive mistake architects are warned about.

Validating at a scale that proves nothing. A design demonstrated on a laptop with twenty requests says nothing about the band you will operate in, and a pilot that cost pennies says nothing about the production bill. This is the quieter mistake, and in AI-era systems it is the one that gets people — because the gap between pilot and production cost is enormous and nobody feels it until the invoice.

The practical habit is cheap: always divide the business number down to a per-second rate before designing anything. It takes thirty seconds and it regularly reveals that the scary-sounding volume is trivial — or that the comfortable-sounding one is not.

Doing the division, and finding the real constraint

The support volume is ~12,000 tickets a month. Stated like that, it sounds like a system that needs serious throughput engineering.

Divided down: 12,000 ÷ 30 ≈ 400 tickets a day. Spread across three time zones with a realistically wide active window of ~16 hours, that is about 25 an hour, or roughly 0.007 requests per second. Even allowing a 10× peak and several model calls per conversation, the platform sits comfortably in the lowest band on the diagram.

That single calculation rules out a great deal of design. No partitioning strategy is needed. No elastic fleet. No queue-based load levelling for throughput reasons. One modest service per path handles this volume with room to spare, which is a strong argument for the simpler end of the style table from Section 1.

Two warnings, though, and they are the reason the exercise is worth doing rather than skipping.

Do not borrow the product’s numbers. Lintel’s product serves ~95,000 daily-active technicians; that is a different system with a different band. Sizing the support platform against the product’s scale would produce an architecture nobody needs, consuming capacity Tomas Beck does not have.

The binding constraints here are not scale at all. They are residency, which decides where inference may run; accuracy, which decides whether the deflection is real; and unit cost, which decides whether the business case closes. A design optimised for throughput would be solving the one problem this system does not have — and would still fail Lena Fischer’s review.

Feeds the workbook

Do the per-second division before you finalise the reference architecture, and write the number in the document. Then check your trade-off table: if any criterion in it is about throughput, confirm it actually discriminates between your options at this scale. If it does not, cut it and replace it with one that does.

→ That closes the module. You have a chosen structure, a reference architecture and a sequence. — Module 4 — The AI Stack

Optional workbook: Exercise 3 turns this module into artifacts. The exercises build on each other, so work them in order or not at all.