TL;DR: You cannot test an AI feature the way you test a login form, because the same input can produce different outputs and several of them can be acceptable. What replaces the usual pass or fail test is an evaluation set, often shortened to "evals": a collection of real inputs, a written definition of a good answer for each, and a score you track every time something changes. Build it from your own data, include the awkward cases, agree the launch threshold with the business before anyone sees results, and keep adding to it from production. If a vendor cannot show you theirs, they are shipping on impressions.
Most AI features look good in a demo. The demo uses inputs someone chose, read by someone who wants it to work. The trouble starts on the first week of real traffic, when customers ask things nobody anticipated, upload documents in the wrong format, or phrase a simple question in a way the system misreads.
An evaluation set is how you find those problems before your customers do, and how you know whether next month's change made things better or worse. This guide is for business owners and product leaders who are commissioning or approving an AI feature and want to know what "we tested it" should actually mean.
We touched on this in choosing an AI model, where an eval set is the right way to compare models, and in software testing and QA, where it sits alongside ordinary tests. This is the longer version.
Why ordinary tests are not enough
A normal software test checks that a given input produces an exact expected output. Click "save" and the record is saved. That works because conventional code is deterministic.
AI features are not. Ask a model to summarise the same lecture twice and you get two different summaries, both of which may be fine. Ask it to classify a support message and it may be right ninety-nine times and confidently wrong on the hundredth. A single test case tells you almost nothing. What you need is a picture of behaviour across many realistic inputs, and a way to see whether that picture is improving or slipping.
What goes into an evaluation set
Real inputs, not invented ones
The examples should come from the actual workload: real customer messages, real documents, real call transcripts, real lecture recordings. Invented examples are almost always cleaner than reality, and an AI feature tuned on clean examples will disappoint on messy ones.
If the product is new and there is no production data yet, collect examples from the closest thing you have: old support tickets, sample documents from early customers, recordings from a pilot.
The cases that cause trouble
A good set is deliberately uncomfortable. Alongside typical inputs, include:
- Ambiguous requests, where a sensible system should ask a clarifying question rather than guess.
- Out-of-scope requests, such as a customer asking a booking assistant for legal advice, where the right answer is a polite refusal or a handoff.
- Bad input, such as blurry scans, half-finished forms, background noise or mixed languages.
- Adversarial input, such as text that tries to make the system ignore its instructions. We cover why this matters for systems that take actions in AI agents with write access.
- High-stakes cases, where a wrong answer costs money, trust or safety.
A written definition of "good"
For each example, someone with business knowledge writes down what a good response looks like. Sometimes that is exact: the invoice total is this number, the caller wants Tuesday at 3pm. Sometimes it is a short checklist: mentions the deadline, does not promise a refund, stays under a paragraph, hands off to a person if the customer is upset.
This step is where most of the value lives, because it forces the team to agree on what the feature is for. Disagreements about the definition of good are much cheaper to resolve here than after launch.
How big it needs to be
Smaller than people expect to start. A few dozen well-chosen examples, read carefully, will surface more real problems than thousands of synthetic ones skimmed by nobody. The set grows over time as production reveals new failure patterns. What matters is coverage of the kinds of input you actually get, not raw size.
How outputs get scored
There are three broad ways to grade, and most real systems use all three.
Automatic checks
Anything that can be checked by code should be. Did the extraction return the right fields? Is the date valid? Did the voice agent call the booking tool with the correct time slot? Is the response under the length limit? Did it include a link it was told never to share? These checks are fast, cheap and run on every change.
Human review against a rubric
Some qualities need a person: is this summary accurate, is the tone right, would a customer find this helpful? The key is a short rubric, so two reviewers grading the same output usually agree. Vague instructions like "rate quality from one to five" produce noise. Specific questions like "does it state any fact not found in the source?" produce signal.
A model grading a model
It is common to use a second AI model to grade outputs against the rubric, because it scales where human review does not. It is useful, and it can be wrong in its own consistent ways. Treat it as an assistant to human review, not a replacement: check its grades against a human sample regularly, and do not let it be the only judge of anything high-stakes.
Agree the launch bar before you see the results
The most common mistake we see is deciding what "good enough" means after looking at the scores. Once a team has seen a number, it is very easy to argue that the number is acceptable.
Instead, agree in advance with the people who own the outcome:
- What score on the typical cases is required to launch.
- Which failures are unacceptable at any rate, such as giving medical or legal advice, quoting a wrong price, or exposing another customer's data.
- What happens when the system is unsure. Often the right design is to hand off to a person, which turns a potential failure into a safe outcome. We wrote about designing that path in voice AI human handoff.
Run it every time something changes
An eval set earns its keep after launch. AI features change in ways ordinary software does not:
- Prompt edits. A small wording change to fix one problem can quietly break three others.
- Model updates. Providers update and retire models. The same model name can behave differently from one month to the next, and you may eventually be forced to switch.
- Retrieval and data changes. If the feature answers from your documents, adding or reorganising content changes the answers. See building an internal AI knowledge base.
- Cost changes. Moving to a cheaper or faster model to control spend is sensible, as we covered in AI cost control, but only if you can show quality held.
Running the set on each of these turns a nervous judgement call into a comparison. The result should be visible to the team, with the previous score next to the new one, so a regression blocks the change instead of reaching customers.
Feed production back into the set
Evals before launch are an educated guess about real traffic. Production is the real thing. The set should keep learning from it:
- Log enough to investigate, within your privacy rules: the input, the output, what the system decided and why.
- Collect signals of failure, such as a customer rephrasing the same question, asking for a person, a staff member correcting a result, or a thumbs down.
- Turn real failures into new eval cases, with the correct answer written down, so the same mistake is caught automatically next time.
For voice systems this is especially important, because calls are fast and unforgiving. With the voice agents we built for CallGuard AI and CallSetter AI, and the multilingual voice and SMS intake behind Fortell AI, realistic replayed calls (interruptions, changed minds, requests the business does not handle) are part of how changes get checked. The production side of that is covered in AI voice agent monitoring.
The same thinking applies outside voice. For an app like LectureNotes AI, which records lectures and turns them into summaries and outlines, the useful questions are concrete: did the summary leave out a key takeaway, or add something the lecturer never said?
What this means for budget and timeline
Building a first eval set is not a large project. It is usually a few days of work inside the build, with most of the effort coming from someone who knows the business writing down what good looks like. The tooling can be simple at first; a spreadsheet of inputs and expected outputs and a script that runs them is a perfectly respectable start.
What it does cost is attention. Somebody on your side has to own the definition of good and review failures regularly. If that is nobody, the set goes stale. The wider approach is in adding AI to an existing product.
Questions to ask the team building your AI feature
- Can you show us the evaluation set, and where did the examples come from?
- Which awkward, out-of-scope and adversarial cases does it include?
- Who wrote the definition of a good answer, and did someone from our business review it?
- How are outputs scored, and where do humans check the automatic grading?
- What launch bar did we agree, and which failures are unacceptable at any rate?
- Does the set run automatically on every prompt, model or data change?
- How do production failures become new eval cases?
- If the model provider changes or retires a model, how quickly can we compare a replacement?
The bottom line
An AI feature is only as trustworthy as the evidence behind it. An evaluation set built from real inputs, with a written definition of good, an agreed launch bar and a habit of running it on every change, is that evidence. It is modest work compared with the build itself, and it is the difference between knowing your AI feature works and hoping it does.
Planning an AI feature and want proof it works before customers see it? Book a demo and we will scope the build and the evaluation set together. See our work: CallGuard AI, CallSetter AI, Fortell AI, LectureNotes AI, Lifemaxxing AI and more.