Test AI answers before launch

A convincing answer in a demonstration is a starting point. Before a customer pilot, a team needs to see how the product behaves when the question is incomplete, the source is out of date, or the answer is not in the collection. A useful test set includes those ordinary situations.

You do not need a large benchmark to begin. You need a small set of questions that represents the job you intend to release, with enough evidence for a reviewer to explain the expected behavior. The first goal is to discover the product’s failure patterns, then decide which ones must be fixed before anyone depends on it.

Gather questions from the actual task

Start with the people who answer these questions today. Ask for recent examples, including the wording people used and the information they left out. Remove private details or use approved fictional substitutes before putting the examples into a test system. Preserve the parts that make the question difficult.

For a product-setup assistant, the set might include inviting a colleague, changing a role, finding an invitation that did not arrive, and asking for an account action that the assistant cannot perform. For an internal procedure tool, it might include current instructions, old terminology, and a question that only a restricted document could answer.

Keep a mix of straightforward and difficult cases. Easy cases show whether the basic path works. Awkward cases show whether the product recognizes its boundaries. A test set made only of questions with clean answers in well-written documents can make an early product look more ready than it is.

Write the expected behavior, not one perfect sentence

For each question, record what a satisfactory answer must contain and what it must avoid. An answer can use different wording and still be correct. A rigid comparison with one model sentence may reject useful responses while overlooking an important missing condition.

Suppose the question is, “Can I invite an outside contractor?” The expected behavior might be to identify the applicable user role, explain the documented invitation path, and ask for clarification if the workspace type changes the answer. The forbidden behavior could be promising that a feature is available on every plan when the source says otherwise.

Include the supporting source and its relevant section. The reviewer should be able to check the expectation without searching the entire knowledge base. Also record when the expectation was last reviewed. An evaluation can become wrong when the product policy changes, just as an answer can.

Separate retrieval from answer writing

When an answer fails, first inspect the material the product used. Did it retrieve the right document? Did it find the relevant section? Was that section current? If the correct source never reached the answer step, changing the response wording may not address the underlying problem.

Next, compare the answer with the retrieved passage. A system can find the correct document and still omit an exception, combine incompatible rules, or turn a suggestion into a requirement. Mark those failures separately so the team can choose a targeted fix.

Microsoft’s Copilot data guidance describes responses grounded in organizational material and citations associated with interactions. Those features make inspection possible, but a reviewer still has to establish whether the source supports the claim. Treat a citation as a route to evidence, not an automatic passing grade.

Add questions with missing information

A customer may ask, “Why can’t I add someone?” without naming the product version, role, or error message. The expected response may be a clarifying question. A test should reward that behavior when the missing detail changes the instructions.

Write several versions of the same question with different amounts of context. One may be answerable immediately, another may need clarification, and another may require an account-level investigation. This reveals whether the assistant is using the information supplied or delivering the same generic steps every time.

Check the quality of the clarification too. Asking five questions when one would resolve the uncertainty creates avoidable work. A good next question should help choose between plausible paths, and the product should remember the answer within the conversation.

Test what is absent and what conflicts

Add a question whose answer does not appear in the approved source material. Decide what the product should do: explain that it cannot confirm the detail, point to a person, or offer the nearest documented information with a clear limit. It should not invent a policy to make the exchange feel complete.

Then create a controlled conflict between two approved test documents. For example, an old guide might describe one approval step while the current procedure describes two. The test should show whether the system follows the intended freshness rule or flags the disagreement for review.

Keep these cases clearly labeled as test fixtures. Do not alter live operational documents merely to create an evaluation. The point is to exercise the behavior in a controlled environment and preserve a record that can be rerun after a change.

Include access and handoff cases

If the product retrieves internal material, run the same question as users with different permissions. Check the answer, the cited document title, and any preview. A restricted detail should not leak through a summary or a suggested follow-up question. Include a case where access has been removed since the source was connected.

For customer support, ask explicitly for a person and test a situation that should trigger escalation. Microsoft’s live-agent handoff documentation illustrates the transfer pattern. In your own product, verify the real destination and the context that arrives there. A button labeled “Contact support” does not by itself prove that a useful handoff occurred.

Also test the unavailable path. If the help desk cannot accept a transfer, the product should describe the next available step accurately. It should not claim that a case was created when the external system rejected the request. That distinction belongs in the evaluation before the first customer sees it.

Use a small review rubric

A first rubric can ask four questions: is the answer supported, does it address the user’s task, does it handle uncertainty appropriately, and does it respect the action boundary? Add a short note for every failed criterion. A reviewer should be able to point to the specific line or behavior that caused the failure.

Anthropic’s guide to evaluating agents discusses examining complete interactions and using defined criteria. For a small team, reading the actual conversation is especially useful. A final answer can look correct even if the preceding exchange collected unnecessary information or sent the user down an incorrect path.

Keep severe failures visible rather than hiding them in an average. An unsupported account action, a privacy leak, and an awkward phrase should not carry identical weight. Decide which failures block the pilot and which can be addressed through a clearly stated limitation or a smaller initial scope.

Compare changes against the same set

When you change a prompt, a source collection, or a model configuration, rerun the established cases. Add the newly discovered failure as a case, but keep the older ones. A fix for one question can make another answer worse, and a stable set gives the team a way to notice that tradeoff.

Record the version tested, the date, and the reviewer’s decision. If behavior varies across runs, repeat the affected cases enough to understand the variation rather than keeping only the most flattering answer. The exact number of runs depends on the task and the consequence of a failure; the record should state what was actually tested.

Before launch, review the remaining failures with the person responsible for the pilot. Agree on the allowed task, the monitoring owner, and the conditions for pausing. The result is not a promise that every future answer will be correct. It is a concrete account of what the team checked, what it learned, and what it is prepared to put in front of users.