AI proof of value (PoV): deciding whether a pilot should scale

What is an AI proof of value, and how does it differ from a proof of concept?

A proof of concept tests whether a key technical function can work. A proof of value goes further: does the proposed system improve a defined task at acceptable quality, cost and operating effort? This is a practical distinction, not a universally certified terminology. Producing a plausible summary is feasibility evidence. Helping intended users complete accepted work within a realistic workflow begins to establish value. Neither result alone proves that a company-wide rollout is justified.

Step one: write a hypothesis that could fail

Replace “AI improves productivity” with a bounded claim: for specified documents and a defined review process, total handling time decreases without increasing critical omissions. Define success, failure and inconclusive results before examining outputs. Small samples or highly variable results may call for further evidence. HM Treasury's guidance emphasises an explicit mechanism connecting an AI intervention to outcomes, testable assumptions and early baselines.[S1] The evaluation should be designed to learn, not to manufacture an approval story.

Step two: establish the current process and alternatives

Measure reading, searching, drafting, review, correction and waiting, preserving differences in task difficulty. Do not compare AI on easy tasks with a human average across all cases. Include a better template, ordinary search or a rules-based approach where relevant. The decision concerns the value of this investment compared with credible alternatives. A model-quality benchmark cannot replace the time and effort required for the entire accepted result, especially when human checking remains necessary.

Step three: make the comparison interpretable

Where practical, allocate comparable tasks to the existing and assisted workflows while balancing experience and difficulty. If the same person performs the same task twice, consider memory and practice effects. When random allocation is unavailable, record task type, staff, timing and concurrent workflow changes. A before-and-after improvement may also reflect training, a new template or a seasonal workload change. Describe these limits rather than attributing every difference to AI. Seek evaluation expertise when investment or consequences warrant stronger causal inference.

Step four: represent the intended operating range

Include routine tasks, boundary cases, questions that should be refused and operational failures. Boundary cases might involve contradictory versions, difficult tables or low-quality scans. Refusal cases include unavailable sources, missing permission and requests outside scope. Failure cases include timeouts, withdrawn documents, invalid output formats and cancellation. Retain input versions, required facts, prohibited errors, scoring rules and evidence for every test. Separate development material from the final acceptance set so repeated tuning does not turn the test into a memorised demonstration.

There is no universal sample size that establishes readiness. A small set can reveal obvious faults, but decisions about success rates and expansion need attention to variability, meaningful effect size and consequences. Record the sampling method. An archive of successful routine tasks alone cannot represent the difficult or failed work the system will encounter after deployment. Show which task categories were absent from the evaluation instead of hiding them in a single average.

Build a sampling frame covering document type, difficulty, input quality and user role. Retain selection and de-identification methods and the reasons for exclusions. If a category lacks evidence, exclude it from the expansion recommendation rather than treating it as implicitly passed. Evaluation use also needs appropriate information rights: a test is not permission to send restricted meeting or customer material to a supplier.

Step five: measure quality, work and cost together

Quality may include factual correctness, whether citations support the claims, critical omissions and appropriate refusal. Workflow measures include time to accepted completion, rework, exceptions and user burden. Costs include inference, retrieval, licensing, maintenance and human effort. NIST Measure offers context-sensitive reference actions for testing, measurement and monitoring.[S2] The metrics here must be defined for the actual task. A public model leaderboard is neither a business-value measure nor evidence that the surrounding service is safe to operate.

Before formal scoring, ask two domain reviewers to assess the same small set of outputs independently and compare disagreements. Distinguish critical factual errors, correctable wording, missing support and appropriate refusal, requiring a passage or rule behind each judgment rather than a general impression. When reviewers disagree, inspect whether the task, rubric or reference answer is ambiguous before a designated role records an adjudication. If the rubric changes, rescore affected outputs from both the manual and assisted conditions; do not revise only the cases the model lost. Keep these changes and unresolved disagreements in the report so readers can see how far the apparent quality difference depends on human judgment as well as system behavior.

Illustrative scenario: a meeting-summary assistant

Imagine testing an assistant that extracts decisions from meeting records. Domain reviewers first identify decisions, responsible roles, deadlines and unresolved issues in each source. Compare unaided and assisted summaries through blind review where possible. A missing deadline must remain missing, and a proposal must not become an approval. Include self-corrections, similar project names, unclear audio and mixed languages. These cases expose consequential mistakes that a general judgement of completeness can miss. The scenario is fictional and claims no AiTOM customer result.

Suppose drafting becomes faster but the assistant repeatedly turns discussion into decisions. Reporting the time saving alone would misrepresent the result. Limit the feature to drafts with source timestamps, retain a chairperson's confirmation and retest both decision accuracy and total review time. The appropriate conclusion may be a narrower use case. An early experiment should not become a universal claim about saved time or a promise that the same benefit will appear across the organisation.

Step six: include adoption and maintenance in the trial

Have intended users inspect sources, compare versions, edit outputs and decline suggestions. Rehearse a source update, a model failure and a return to the earlier process. Observe who takes responsibility and how recovery works. If every small issue requires the original developer, record an operating dependency. Maintenance that remains invisible in a tiny trial can dominate after expansion. User satisfaction is useful feedback but does not replace evidence about accepted work and sustainable support.

Step seven: expand, narrow, investigate or stop

Expansion should satisfy quality gates, the value hypothesis and operating prerequisites together. Narrowing is appropriate when some tasks benefit but others remain unsuitable. Further investigation fits insufficient samples or inconsistent comparisons. Stopping should preserve the reason and reusable outputs, rather than adding features to obscure the failed hypothesis. Report configuration, scope, sampling, method, distributions, failure cases, costs and untested conditions. Record a new decision when later evidence changes the recommendation.

Which results are easiest to overinterpret?

A successful demonstration, a high average score or user enthusiasm cannot independently establish lasting value. Average handling time may conceal rare but serious failures. Inspect task categories and poor-performing tails. Model, source and workflow changes can invalidate the earlier result, so rerun affected evaluations. Where training and tooling change together, acknowledge that their separate effects have not necessarily been identified. A proof of value remains evidence within a boundary, not a permanent licence to expand.

Must a proof of value use production data and systems?

Not initially. Offline evaluation can establish basic function and quality before a controlled trial with real users. A claim of full operating value eventually requires evidence of actual use and maintenance, however. Laboratory results alone cannot establish the sustainability of the complete service.

Why is high answer accuracy insufficient for launch?

Access boundaries, refusal, updates, external actions and recovery may remain untested. Correct answers can also impose excessive review effort. Launch acceptance concerns the whole service and its accountable operation; model accuracy is one component of that decision.

How should results be communicated publicly?

Publish only authorised results with traceable methods and evidence. State the task, sample, period, version and limitations. Mark illustrative assumptions clearly. Without representative evidence, avoid claims that the result applies to all companies or guarantees a fixed improvement percentage.

Sources and evidence boundaries

1. S1 · Guidance on the Impact Evaluation of AI Interventions

   HM Treasury / Evaluation Task Force · Accessed 2026-10-22

   Supports the cited principle; operational steps and illustrative scenarios are editorial recommendations rather than reported customer results.

2. S2 · AI RMF Playbook — Measure

   NIST AIRC · Accessed 2026-10-22

   Supports the cited principle; operational steps and illustrative scenarios are editorial recommendations rather than reported customer results.