The eval suite is the product

A client asked us to improve their classification accuracy. Reasonable request. The problem was that nobody could tell us what the current accuracy was, because there was no set of examples everyone agreed on.

We spent the first eleven days building that set. No model work at all. By the end we had two hundred labelled cases, a disagreement rate between their own experts of fourteen percent, and — more usefully — a written definition of what the fourteen percent disagreed about.

Why this is the actual deliverable

The model will be replaced. Probably within a year, possibly by something from a provider that does not exist yet. The eval suite outlives every model you put behind it, and it is the only artefact that lets you swap one for another without a leap of faith.

Teams that treat evaluation as a phase end up rebuilding it every time. Teams that treat it as the substrate ship model changes on a Friday.

What a suite has to contain

• Cases that used to fail. Every production incident becomes a test. This is the cheapest source of high-value cases and almost nobody harvests it.
• Cases at the boundary. The ones your experts disagreed about. These are where the metric is doing real work.
• Cases that must never regress. A small, sacred set. If these break, the change does not ship regardless of how good the aggregate looks.
• A cost and latency budget. Accuracy without a budget is a number that will eventually bankrupt the feature.

The uncomfortable part

Building the suite forces the organisation to decide what correct means, in writing, with names attached. That conversation is genuinely difficult and it is usually avoided by going straight to model work.

Going straight to model work does not remove the conversation. It defers it until you are in production and the stakes are higher.