How to tell if your AI actually works

Here's a conversation that happens in most teams building with AI, usually around week six.

"The new prompt feels better." "Does it? The old one seemed fine to me." "No, look at this example." "Sure, but try this other one."

Both people are right, and neither can prove anything, because nobody has a number. So the decision gets made by whoever is more senior or more tired, and the feature ships on a hunch. Then next month someone changes the prompt again and the whole conversation repeats.

The fix is unglamorous and takes an afternoon: write down what "working" means, before you argue about whether it works.

What an evaluation set actually is

A list of inputs and what a good output looks like. That's it.

Thirty rows in a spreadsheet is a real evaluation set. It doesn't need a framework, a platform, or a vendor. What it needs is to exist, and to have been written down before you started tuning, because a test you invent after seeing the results is a test you've already passed.

The reason this works isn't sophistication. It's that it converts "feels better" into "23 of 30 versus 19 of 30," and that's a sentence two people can agree about.

Start by writing down what wrong looks like

Most teams stall here because "correct" is hard to define for open-ended output. There's no single right summary, no single right answer.

So don't start with correct. Start with wrong, that's much easier and it's what you actually care about.

Sit down and list the ways your feature could fail in a way that matters:

• It invents a policy that doesn't exist.
• It answers in three paragraphs when the user wanted a sentence.
• It says "I don't know" when the answer was right there.
• It's rude, or oddly casual, or apologises four times.
• It leaks something from another customer's data.

Now you have categories. And most of them are checkable without any judgement at all, length, format, whether a forbidden phrase appeared, whether it cited a real document. The subjective ones you can score by hand, because there are thirty of them and it takes twenty minutes.

This inversion is the whole trick. "Is it good?" is unanswerable. "Did it do any of these seven bad things?" is a checklist.

Get real inputs, not invented ones

The most common way an evaluation set lies to you is that you wrote the questions yourself.

You know the system. You know what it's called, what the documents say, what vocabulary it expects. Your questions unconsciously fit it. They pass, and then real users arrive asking things in words the system has never seen, and everything falls over.

So: use real inputs if you have any. Support tickets, search logs, whatever people actually typed. If you don't have real inputs yet, get someone who has never seen the system to write twenty questions. Take them to lunch. It's the highest-value twenty minutes in the project.

And include the horrible ones. The ambiguous question. The one where the answer isn't in your documents. The one that's slightly hostile. Real traffic contains all of these and your happy-path spreadsheet doesn't.

Thirty examples is enough to start

Not because thirty is statistically comfortable, it isn't, but because thirty exists and three hundred doesn't.

An evaluation set of thirty real cases catches the failures that matter: the format is wrong, the retrieval misses, the tone is off, the model hedges on things it should answer. Those show up loudly and immediately. You do not need a large sample to notice that a third of your answers are the wrong shape.

What thirty won't do is resolve small differences. If version A scores 22 and version B scores 23, that's noise. Don't ship on it, and don't let anyone claim it as a win. Small samples give you loud signals and nothing else, which is fine, the loud signals are where the value is early on.

Grow it when it stops being useful. Every real failure someone reports becomes a new row. In six months you'll have two hundred cases, all of them derived from things that actually went wrong, which is a far better set than two hundred you invented on day one.

Don't tune against the set you're measuring with

This is the same discipline as a train/test split, and it gets broken constantly.

If you tweak the prompt, check the score, tweak again, check again, twenty times, your score is no longer measuring quality. It's measuring how well you've fitted your own test. That's not a hypothetical: each check is a comparison, and enough comparisons guarantee an optimistic result by chance.

The fix is boring. Split your cases. Use most of them for iteration. Keep a handful you look at once, at the end, before shipping. If those hold up, you learned something real. If they collapse while your tuning set looks great, you learned something even more valuable.

Almost nobody does this, and it's why "it scored well in testing" and "it works in production" keep turning out to be different claims.

Why nobody builds this

The advice above is not controversial and is still rarely followed, which is worth examining rather than repeating the advice more firmly.

Building an evaluation set is unrewarding work. It produces no demo, it takes a day or two of attention that could go into shipping features, and its entire value is preventing a future problem that has not happened yet. The counterfactual is invisible: nobody thanks you for the regression that did not reach production.

It also requires deciding what correct means, which is uncomfortable. Teams often discover during this exercise that they do not agree on what the system is for, and that disagreement is easier to leave unexamined. Writing down thirty examples with expected outputs forces the argument into the open, which is valuable and unpleasant in the same moment.

The practical answer is to make it small enough that it happens. Thirty examples in a spreadsheet, scored by hand the first time, and run more than once, beats a proper evaluation framework that stays on the backlog. The perfect version of this never gets built; the crude version pays for itself the first time a prompt change quietly breaks something.

What an evaluation set cannot tell you

Being clear about the limits matters, because a scoreboard invites more confidence than it deserves.

It measures the cases you thought of. A curated set is a sample of your imagination, not of reality. Real inputs are stranger, longer, more ambiguous and more adversarial than anything you would write yourself, which is why the advice to draw examples from actual traffic rather than inventing them is not a nicety.

It goes stale silently. Usage patterns move, users learn what the system responds to well, and the distribution of what arrives shifts away from what you froze months ago. A set that no longer resembles live traffic will report stability while quality falls.

It rewards what it can score. Anything easy to measure gets optimised and anything hard to measure gets ignored, which is Goodhart's law arriving on schedule. If tone, appropriateness and knowing when to decline are not in the set, they will drift, and the score will not move.

It says nothing about the tail. Averages hide the failures that matter. A system that is right ninety-five percent of the time and catastrophically wrong in one specific situation scores well and is dangerous, and the only way to find that situation is to look at individual failures rather than at the aggregate.

The right posture is that an evaluation set is a regression guard, not a quality measure. It tells you reliably when something got worse. It tells you much less than it appears to about whether the thing is good.

What to measure

Depends on the task, and the honest answer is that the metric you want is usually simpler than the one you'll be sold.

For extraction and classification, it's just accuracy, and you should break it down. Overall accuracy hides everything: a system that's 95% right overall and 40% right on the rare category that matters is a system that doesn't work. Look at it per category. Always.

For retrieval, recall. Of the questions where the answer exists in your corpus, how often did the right passage come back? This is the number that caps your entire RAG system, and it's the one people don't measure, which is why they spend weeks blaming the model.

For generated text, resist the automated similarity scores. BLEU, ROUGE and their relatives measure word overlap with a reference answer, and word overlap is not quality. A perfect rewrite scores badly. A fluent lie scores well. Use a checklist instead: did it answer the question, did it stay in the source, did it match the format, was the tone right. Score by hand on thirty cases. It's more honest than a number that correlates with nothing you care about.

For anything user-facing, remember the metric is a proxy. You optimise the checklist; you care about whether people got what they needed. Those diverge, and the checklist won't tell you when.

Using a model to grade the model

This works better than it sounds and worse than it's sold.

Asking a model to judge outputs, "does this answer the question, yes or no?", is useful for scale. It catches obvious failures reliably, it's fast, and it costs pennies. If you have three hundred cases and no appetite for reading them, it's the only practical option.

But the judge is a model, with all the properties you deployed it to check. It has preferences. It likes longer answers, it likes confident ones, and it agrees with itself more than it should. So a model-graded score can drift upward while quality doesn't move at all.

The workable compromise: let the model grade everything, then read twenty by hand yourself and see whether you agree with its verdicts. If you do, trust it for the rest. If you don't, you've learned that your judge needs work, which is worth knowing before you make decisions on its output.

The scoreboard beats the argument

Here's what actually changes once you have thirty rows and a number.

Prompt changes stop being a matter of taste. Somebody says "I think we should say X instead of Y," and instead of a debate you run it and find out. Most suggestions turn out to do nothing, which is useful information and impossible to get any other way.

Model upgrades become a decision instead of an assumption. The new model is better on benchmarks; is it better on your thirty? Sometimes no. Sometimes it's better and four times the price and the old one was fine. You cannot know without the set.

And regressions become visible. Without an evaluation set, quality degrades silently, someone tweaks a prompt, something drifts, and you find out from a customer three weeks later. With one, you find out in a minute.

Start today, badly

The mistake here isn't picking the wrong metric. It's waiting until you have time to do this properly, which is a time that does not arrive.

Open a spreadsheet. Two columns: what someone asked, what a good answer looks like. Fill in fifteen rows from real questions. Run them. Count how many are acceptable.

That number is now the most valuable thing in your project. It's the thing that turns "I think it's better" into "it went from 9 to 14," and it's the difference between improving a system and moving it around.

Everything else in this piece is refinement. The spreadsheet is the whole idea.

The concepts behind this: train/test split, overfitting, hallucination and prompt engineering, each explained at five levels from plain English to the research frontier.

The short version

Evaluating an AI system means measuring its quality with a fixed, representative set of test cases scored consistently, rather than trying it a few times and forming an impression. Informal spot-checks are anecdotes, biased by which examples you picked and how you read them, and they miss rare failures. A real evaluation set, even fifty to a hundred tasks drawn from your actual use case, turns it feels better into a stable number you can compare across versions. Automated grading, including using one model to judge another against a rubric, scales this up, though it carries biases and works best calibrated against human review on the cases that matter. Evaluation is the difference between knowing an AI system works and hoping it does, and it starts with a fixed scoreboard of real tasks, not a few impressive demos.

Common questions

What's the fastest way to start evaluating an AI system? Write down ten to twenty real inputs with the answers you'd accept, and run them on every change. A small, fixed, honest test set beats an elaborate framework you never finish, start today, badly, and improve it as failures teach you what to measure.

Is using a model to grade another model reliable? LLM-as-judge is useful and scalable, but it inherits biases, toward length, toward its own style, toward whichever answer came first. Treat its scores as a signal to track over time, not ground truth, and calibrate it against human judgement on a sample.

Why does a fixed scoreboard matter more than argument? Because "it feels better" doesn't survive contact with a second opinion. A stable set of test cases with tracked pass rates turns vibes into a number you can defend, compare across versions, and watch for regressions.

Why can't you evaluate an AI system by just trying it a few times? Because informal spot-checks are anecdotes, not measurement. Trying a system a handful of times tells you how it did on those exact inputs, not how it performs across the range of real cases, and it is heavily biased by which examples you picked and how you read the outputs. Small samples also miss rare but important failures. A real evaluation uses a fixed set of representative test cases scored consistently, so you get a stable number you can compare across versions, rather than a shifting impression that changes with your mood and memory.

What is an evaluation set? An evaluation set is a fixed collection of test cases, drawn from your actual use case, that you run a system against to measure its quality. Each case has an input and a way to judge the output, whether an expected answer, a rubric, or a grader. Because the set stays fixed, scores are comparable across model versions and changes, turning it feels better into a number you can track. A focused set of even fifty to a hundred real tasks is often more informative than any public benchmark, because it measures performance on the work you actually care about.

What is LLM-as-a-judge, and is it reliable? LLM-as-a-judge uses one language model to grade the outputs of another against a rubric, which scales evaluation far beyond what human graders can review by hand. It works reasonably well and agrees with human judgment often enough to be useful, but it has known biases: it can favour longer answers, prefer outputs similar to its own style, and be swayed by position or phrasing. Used carefully, with a clear rubric, awareness of these biases, and calibration against some human-graded examples, it is a practical screening tool. It should complement human review on the cases that matter most, not fully replace it.

How many test cases do you need to evaluate an AI system? Fewer than people expect to start, more than one might hope for confidence. A focused set of fifty to a hundred real, representative tasks is enough to catch major differences between versions and turn vague impressions into a usable score. As stakes rise, you expand coverage to include edge cases, known failure modes, and the specific situations that matter for your application. The point is representativeness, not raw size: a hundred cases that mirror your real workload tell you more than a thousand generic ones, because they measure the performance you actually depend on.