AI system evaluation: assessing quality with test sets and evidence

How should an AI system be evaluated before release?

Turn real tasks into tests with inputs, environment, expected outcomes, failure conditions and grading evidence. Repeat trials and inspect actual outputs and state. Fluent text is not necessarily correct, a completion claim is not evidence of an action, and one successful attempt does not establish reliability. The method here provides no universal passing percentage or zero-risk guarantee. Models, instructions, tools, information, permissions and environment form the tested system. A public model ranking cannot substitute for evaluating that configuration in its intended use.

Step one: define an assessable task contract

For a question about travel policy, identify the current policy, readable scope, required conditions, source references and expected response when information is missing. For an action, also specify permitted changes and prohibited side effects. NIST's Measure guidance calls for operating conditions, validity and limitations beyond those conditions.[S1] Ask knowledgeable people to judge the same case before automating assessment. If they cannot agree on what constitutes success, clarify the task. Evaluation should not punish a system for an unstated assumption or conceal a disagreement about the underlying business rule.

Step two: build representative and challenging cases

Use permitted, appropriately de-identified questions, historical failures, frequent tasks and designed boundaries. Include normal work, missing or conflicting information, expiry, unauthorised requests, tool failure and malicious material. Keep provenance and expected behaviour. Separate development cases from held-out assessment so that improvement is not merely memorisation. Present languages, roles and information volumes separately. Many easy cases must not dilute a serious failure category. Dataset size depends on purpose, variability and consequence; a fixed count cannot guarantee coverage. Maintain cases when source policies change so that an outdated reference answer does not penalise correct current behaviour.

Step three: make each case reproducible

Record actor, task, input, readable information, starting state and versions. Define required answers, acceptable variants, refusal conditions and prohibited actions. Assign the relevant code checks, expert criteria and source comparisons, with a known reference result where feasible. Capture output, tool requests and responses, actual state, time, cost, failure category and retry information. Test data needs governance too: confidential material must not become an unrestricted benchmark for convenience. A later reviewer should understand why a case passed without relying on a private recollection of what the author intended.

Step four: choose and calibrate graders

Anthropic distinguishes code-based, model-based and human grading and emphasises calibration of model grading with people.[S2] Code suits format, files, denied access and observable state. Meaning and domain judgement need explicit criteria and suitable human expertise. A model may assist with large-scale preliminary review but can favour longer answers, misread citations or accept a false premise. Compare model and human judgements on the same cases and preserve disagreements. Let an uncertain grader report uncertainty. Agreement between generated opinions is not a replacement for directly verifiable evidence.

Step five: verify results and necessary process constraints

After tool execution, inspect the file, record or service state. Do not rely on the final statement that work is complete. A trace can reveal forbidden access, skipped approval or repeated actions that a correct-looking result hides. Outcome success does not cancel an authority violation. Conversely, do not require one exact tool sequence unless safety or the task contract makes it necessary. Another valid method may achieve the same acceptable result. Distinguish essential controls from incidental implementation choices so that the suite detects failure without rejecting harmless improvements.

Step six: repeat trials under recorded conditions

Keep relevant information, tool conditions, authority and versions stable. Use isolated, recoverable environments so that leftover files or state do not help a later trial pass. Report successes and total trials for each task, distinguishing at least one success after retries from dependable repeated performance. Label timeouts and infrastructure errors explicitly, with their scoring treatment and effect on users. Do not automatically blame every failed run on the model or discard all operational failures. They are part of understanding the system's actual reliability even when a separate analysis identifies their source.

Hypothetical scenario: the answer is right but cites restricted information

Imagine an employee receives correct travel guidance with a restricted attachment from another department. Answer similarity alone might produce a high score. A complete check examines readable scope for that identity, citation access and the expected denial. Another case removes the authorised source and expects an honest insufficient-information response rather than a plausible answer from memory. These scenarios test content and authority together. They do not establish that one passing case covers every role, document or version. Keep the conclusion tied to what was actually exercised.

Step seven: separate capability, regression and release gates

Capability tests explore difficult work. Regression tests protect behaviour that previously worked. Consequential authority or safety failures may have gates that cannot be averaged away. After a version change, report new successes, regressions, untested areas and known limitations. A changed grader needs appropriate independent review and retained historical results, especially if it turns a failure into a pass. Human acceptance of an exploratory result is not blanket permission for automatic external action. Release decisions depend on intended use and consequences, not a single attractive benchmark number.

Define measures without hiding failure

Task success can mean trials passing all required criteria divided by valid trials, accompanied by category-level results and reasons. Citation support checks whether sources support specific claims, not only whether links exist. Evaluate both cases that should refuse and cases that should answer so that universal refusal cannot look successful. Include failed retries and human review in cost, and distinguish active work from waiting. Report sample size and variability. One overall score does not establish complete quality or business value, and a high average can coexist with an unacceptable rare failure.

Where simpler evaluation is insufficient

Professional, high-consequence and fairness questions may require domain experts, appropriate data and a distinct study. Offline results may not represent live traffic, information distribution or dependencies. Monitor new failures after release and add confirmed cases to regression without leaking data or contaminating held-out tests. A failed result may reveal a system problem, ambiguous requirement or faulty grader; investigate before changing the model. Removing difficult cases to improve the score undermines the evidence the evaluation is meant to provide. Keep known limitations available to the people making operational decisions.

Frequently asked questions

How many cases are enough? Cover core work and serious failure modes first, then expand according to variation and risk; there is no universal magic number. Can AI judge everything? It can assist, but needs calibration with people and observable facts. Must every small change rerun everything? Select necessary regression from the impact, widening coverage for shared instructions, information or tools. Without current evidence, a historic pass cannot establish the behaviour of the changed version. The aim is a maintained evidence system rather than a one-time demonstration.

Sources and evidence boundaries

1. S1 · NIST AI RMF Playbook: Measure

   NIST · Accessed 2026-10-22

   Supports the cited principle; operational steps and illustrative scenarios are editorial recommendations rather than reported customer results.

2. S2 · Demystifying evals for AI agents

   Anthropic · Accessed 2026-10-22

   Supports the cited principle; operational steps and illustrative scenarios are editorial recommendations rather than reported customer results.