How to tell if your AI actually works

Here's a conversation that happens in most teams building with AI, usually around week six.

"The new prompt feels better." "Does it? The old one seemed fine to me." "No, look at this example." "Sure, but try this other one."

Both people are right, and neither can prove anything, because nobody has a number. So the decision gets made by whoever is more senior or more tired, and the feature ships on a hunch. Then next month someone changes the prompt again and the whole conversation repeats.

The fix is unglamorous and takes an afternoon: write down what "working" means, before you argue about whether it works.

What an evaluation set actually is

A list of inputs and what a good output looks like. That's it.

Thirty rows in a spreadsheet is a real evaluation set. It doesn't need a framework, a platform, or a vendor. What it needs is to exist, and to have been written down before you started tuning — because a test you invent after seeing the results is a test you've already passed.

The reason this works isn't sophistication. It's that it converts "feels better" into "23 of 30 versus 19 of 30," and that's a sentence two people can agree about.

Start by writing down what wrong looks like

Most teams stall here because "correct" is hard to define for open-ended output. There's no single right summary, no single right answer.

So don't start with correct. Start with wrong — that's much easier and it's what you actually care about.

Sit down and list the ways your feature could fail in a way that matters:

• It invents a policy that doesn't exist.
• It answers in three paragraphs when the user wanted a sentence.
• It says "I don't know" when the answer was right there.
• It's rude, or oddly casual, or apologises four times.
• It leaks something from another customer's data.

Now you have categories. And most of them are checkable without any judgement at all — length, format, whether a forbidden phrase appeared, whether it cited a real document. The subjective ones you can score by hand, because there are thirty of them and it takes twenty minutes.

This inversion is the whole trick. "Is it good?" is unanswerable. "Did it do any of these seven bad things?" is a checklist.

Get real inputs, not invented ones

The most common way an evaluation set lies to you is that you wrote the questions yourself.

You know the system. You know what it's called, what the documents say, what vocabulary it expects. Your questions unconsciously fit it. They pass, and then real users arrive asking things in words the system has never seen, and everything falls over.

So: use real inputs if you have any. Support tickets, search logs, whatever people actually typed. If you don't have real inputs yet, get someone who has never seen the system to write twenty questions. Take them to lunch. It's the highest-value twenty minutes in the project.

And include the horrible ones. The ambiguous question. The one where the answer genuinely isn't in your documents. The one that's slightly hostile. Real traffic contains all of these and your happy-path spreadsheet doesn't.

Thirty examples is enough to start

Not because thirty is statistically comfortable — it isn't — but because thirty exists and three hundred doesn't.

An evaluation set of thirty real cases catches the failures that matter: the format is wrong, the retrieval misses, the tone is off, the model hedges on things it should answer. Those show up loudly and immediately. You do not need a large sample to notice that a third of your answers are the wrong shape.

What thirty won't do is resolve small differences. If version A scores 22 and version B scores 23, that's noise. Don't ship on it, and don't let anyone claim it as a win. Small samples give you loud signals and nothing else, which is fine — the loud signals are where the value is early on.

Grow it when it stops being useful. Every real failure someone reports becomes a new row. In six months you'll have two hundred cases, all of them derived from things that actually went wrong, which is a far better set than two hundred you invented on day one.

Don't tune against the set you're measuring with

This is the same discipline as a train/test split, and it gets broken constantly.

If you tweak the prompt, check the score, tweak again, check again — twenty times — your score is no longer measuring quality. It's measuring how well you've fitted your own test. That's not a hypothetical: each check is a comparison, and enough comparisons guarantee an optimistic result by chance.

The fix is boring. Split your cases. Use most of them for iteration. Keep a handful you look at once, at the end, before shipping. If those hold up, you learned something real. If they collapse while your tuning set looks great, you learned something even more valuable.

Almost nobody does this, and it's why "it scored well in testing" and "it works in production" keep turning out to be different claims.

What to measure

Depends on the task, and the honest answer is that the metric you want is usually simpler than the one you'll be sold.

For extraction and classification — it's just accuracy, and you should break it down. Overall accuracy hides everything: a system that's 95% right overall and 40% right on the rare category that matters is a system that doesn't work. Look at it per category. Always.

For retrieval — recall. Of the questions where the answer exists in your corpus, how often did the right passage come back? This is the number that caps your entire RAG system, and it's the one people don't measure, which is why they spend weeks blaming the model.

For generated text — resist the automated similarity scores. BLEU, ROUGE and their relatives measure word overlap with a reference answer, and word overlap is not quality. A perfect rewrite scores badly. A fluent lie scores well. Use a checklist instead: did it answer the question, did it stay in the source, did it match the format, was the tone right. Score by hand on thirty cases. It's more honest than a number that correlates with nothing you care about.

For anything user-facing — remember the metric is a proxy. You optimise the checklist; you care about whether people got what they needed. Those diverge, and the checklist won't tell you when.

Using a model to grade the model

This works better than it sounds and worse than it's sold.

Asking a model to judge outputs — "does this answer the question, yes or no?" — is genuinely useful for scale. It catches obvious failures reliably, it's fast, and it costs pennies. If you have three hundred cases and no appetite for reading them, it's the only practical option.

But the judge is a model, with all the properties you deployed it to check. It has preferences. It likes longer answers, it likes confident ones, and it agrees with itself more than it should. So a model-graded score can drift upward while quality doesn't move at all.

The workable compromise: let the model grade everything, then read twenty by hand yourself and see whether you agree with its verdicts. If you do, trust it for the rest. If you don't, you've learned that your judge needs work — which is worth knowing before you make decisions on its output.

The scoreboard beats the argument

Here's what actually changes once you have thirty rows and a number.

Prompt changes stop being a matter of taste. Somebody says "I think we should say X instead of Y," and instead of a debate you run it and find out. Most suggestions turn out to do nothing, which is genuinely useful information and impossible to get any other way.

Model upgrades become a decision instead of an assumption. The new model is better on benchmarks; is it better on your thirty? Sometimes no. Sometimes it's better and four times the price and the old one was fine. You cannot know without the set.

And regressions become visible. Without an evaluation set, quality degrades silently — someone tweaks a prompt, something drifts, and you find out from a customer three weeks later. With one, you find out in a minute.

Start today, badly

The mistake here isn't picking the wrong metric. It's waiting until you have time to do this properly, which is a time that does not arrive.

Open a spreadsheet. Two columns: what someone asked, what a good answer looks like. Fill in fifteen rows from real questions. Run them. Count how many are acceptable.

That number is now the most valuable thing in your project. It's the thing that turns "I think it's better" into "it went from 9 to 14," and it's the difference between improving a system and moving it around.

Everything else in this piece is refinement. The spreadsheet is the whole idea.

The concepts behind this: train/test split, overfitting, hallucination and prompt engineering — each explained at five levels from plain English to the research frontier.