AI That Cannot Be Tested Cannot Be Trusted: Building an Enterprise Evaluation System

Published: at 03:05 AMSuggest Changes

The product team has shown an AI assistant handling familiar questions. Operations likes the speed. A senior executive tries two prompts, gets sensible answers and asks when it can go live. Then risk asks what happens when policy changes, sources conflict, or a customer asks for something the system must not promise. The room has enthusiasm. It does not yet have evidence.

That is the tension CIOs need to resolve. Every week spent waiting for perfect certainty leaves useful work on the table. Every release based on a demo turns confidence into an unmanaged operating risk. The answer is an evaluation system that tells leaders what an AI service can do, where it must stop, and whether a change has made it better or worse.

NIST places measurement inside lifecycle risk management, alongside governance, mapping and management. Evaluation cannot be a final quality check after product and workflow decisions have been made. It reveals the limits of the service the organisation intends to operate. NIST AI 600-1

Begin with the decision at risk

The model is rarely the unit that needs trusting. The business process is.

A knowledge assistant drafting internal HR guidance, a system classifying potential fraud, a coding agent preparing a pull request and an agent responding to a customer all produce language. Their failure modes are radically different. The HR assistant must draw on the current policy, cite it where appropriate and signal uncertainty. A fraud workflow needs a low tolerance for missed high-risk signals and a dependable escalation path. A coding agent must remain inside authorised components, satisfy relevant tests and preserve security and performance constraints.

Write the job down as an observable contract before selecting a benchmark or setting a score:

Many programmes become vague here. Teams measure fluency, response time or token cost because those measures are readily available, while leaving the decision that matters unexamined. Product, operations, engineering and risk should agree the contract. The workflow owner must accept residual business risk.

A service assistant may answer routine delivery questions without review. Disputed charges, cancellation requests from vulnerable customers and claims that conflict with the order record should reach a trained colleague. Evaluation must show that the assistant retrieved the right record, recognised the boundary and supplied evidence for the reviewer.

Build evidence from the work people actually do

The first durable asset is a golden set: representative cases with agreed expectations. It is not a generic benchmark downloaded from the internet. It is a versioned collection of real work selected with people who understand the process, including the awkward cases that make the process expensive or risky.

For a support assistant, include ordinary questions, incomplete requests, conflicting account information, obsolete policies, sensitive-data requests and matters that must reach a human. For a coding agent, include contained fixes, misleading issue descriptions, missing acceptance criteria, changes crossing a service boundary and tasks it should decline because it lacks the required context.

Start small. OpenAI’s November 2025 guidance recommends reviewing 50 to 100 outputs from an early system to reveal failure patterns. That is vendor guidance, not a universal sample-size rule. Assemble an initial set, examine results with the people accountable for the work, then add cases where the service fails or the workflow changes. OpenAI

Each case needs an assessment method that matches the task. Some checks are deterministic: a classification matches an approved label, a required field is present, a cited policy is current, or an action stayed within permission. Others require calibrated expert judgement. A domain reviewer may assess whether a recommendation is complete, supported by evidence and appropriate for the facts. The rubric needs enough precision for two reviewers to explain disagreement. “Looks reasonable” is an opinion; it is not a control.

Treat the set as product infrastructure. Give every case an owner, a reason for inclusion, expected behaviour and a version. Keep the source documents and policy versions available to reviewers. Remove cases that no longer represent live work. When a production incident exposes a new failure, turn it into a case before the memory of the incident fades.

Test the path to the answer

A polished answer can still leave the enterprise exposed.

An agent may reach the right conclusion after querying an unauthorised source. A retrieval assistant may cite a plausible policy that was superseded last month. A customer-service tool may write a warm response that grants a refund it lacks authority to approve. Final-text scoring misses the behaviour that caused the risk.

A credible evaluation system therefore separates several forms of evidence.

Task quality asks whether the answer, classification, recommendation or completed action meets the defined standard. Golden cases and expert rubrics belong here.

Grounding and provenance asks whether the service used approved, current sources and whether a reviewer can trace a material answer to its evidence. A correct answer without a reliable basis is fragile the moment policy changes.

Safety and control asks whether the service refuses prohibited requests, withstands hostile or irrelevant instructions, respects data boundaries and escalates work outside its authority. Include adversarial cases deliberately. Ordinary examples do not reliably expose unsafe behaviour.

Tool and workflow behaviour asks what happened across the full path. Did the agent call an allowed tool? Did it retry pointlessly? Did a policy check stop an unsafe action? Did the reviewer receive the right evidence at the right moment? This is the layer that matters when an AI service moves from drafting to changing records or triggering work.

Service qualities cover latency, availability, cost, accessibility and recovery. A service that answers accurately after five minutes, overruns its cost envelope or fails without a usable fallback has not met its operating obligation.

Anthropic makes the architectural point clearly in its guidance on agent evaluation: the model and its harness must be evaluated together. Its emphasis on specified tasks, stable test environments and thorough testing for coding agents applies equally to other enterprise services. A model score says little about the reliability of the system built around it. Anthropic

Put evaluation in the release path

A test suite kept in a spreadsheet is a record of good intentions. It becomes a control only when it governs change.

A material alteration to the model, prompt, retrieval index, tool schema, policy rule or permission scope should trigger relevant evaluation before users see it. The gate should reflect risk. A wording change to an internal drafting assistant deserves a lighter review than a modification to a tool-using workflow that can change customer records or inform a regulated decision. Lighter must still mean explicit.

Set a release baseline for each service. It may include a minimum task-quality threshold, zero tolerance for named policy breaches, a maximum unsupported-claim rate, a latency budget and mandatory human review for a defined class of case. Resist the temptation to collapse these into one composite score. A system that becomes slightly more fluent while missing high-severity fraud indicators has declined in quality, whatever the average says.

Keep comparisons honest. Run a candidate configuration and the live configuration against the same versioned set. Record the model, prompts, tools, data sources, graders and results. Investigate case-level changes rather than accepting an aggregate percentage as a verdict. The leadership team needs a concise release view; the delivery team needs to know which cases moved, why they moved and whether the difference is acceptable.

This also changes vendor management. A provider upgrade is a production change, even if the endpoint name remains the same. So is a new retrieval corpus, an updated policy prompt or a modified tool permission. If the organisation cannot compare behaviour before and after, it has chosen to discover the difference through customers and operations.

Production supplies the next tests

Offline evidence establishes the starting position. Live work produces the rest of the evidence.

Instrument both technical behaviour and the outcome that justified the service. Monitor quality samples, human overrides, escalations, abstentions, tool failures, retrieval misses, retry rates, latency, cost per completed task and the relevant business measure. A service workflow may track repeat contacts and resolution quality. An engineering workflow may track review rework, defect escape and rollback rate. An internal knowledge assistant may track whether employees complete the task without searching elsewhere or escalating an incorrect answer.

Sample real outputs for expert review, particularly for high-impact cases and after material changes. Do not wait for a formal incident. A rising correction rate, a cluster of unsupported answers or an unusual tool-call pattern is early evidence that the golden set needs a new case and the service needs attention.

The UK Government’s assurance guidance distinguishes testing the whole AI system from evaluating the model and calls for evidence through planning, development, deployment and monitoring. That is the right operating frame. Production feedback should become test cases, release criteria and workflow changes. It should not disappear into a dashboard that nobody reads until an audit or an incident. UK Government

Make ownership visible

The product or workflow owner defines good work and accepts residual business risk. Domain experts curate and review cases. Engineering and platform teams make evaluations repeatable in delivery and observable in production. Security, privacy and risk functions specify control tests and evidence needs. Independent assurance can challenge the service, but it cannot own daily quality for a business process it does not run.

Fund this work inside the product. Cases go stale, policies change, suppliers release upgrades and users find new routes to failure. The evaluation system needs a review cadence, versioned assets and clear decision rights.

Here is the leadership test: when an executive asks whether an AI service is ready for a broader scope, can the accountable owner show the work it was tested against, the boundaries it will enforce, the evidence behind its decisions and the conditions that will stop or roll it back? If the answer is a demo, the service is still an experiment. If the answer is evidence, the enterprise has something it can operate.

Sources

Previous Post
The Agentic AI Cost Problem: Why Token Spend Is Not the Unit That Matters
Next Post
Choose Your First Agentic Workflow by Reversibility, Not by Excitement