How to evaluate an AI agent for business data

Evaluate an AI agent against the business job it is expected to perform, not against a generic model benchmark. A production test should use representative questions and measure correctness, evidence, source selection, authorization, task completion, operational behavior, and recovery when something goes wrong.

An intelligent-sounding model is irrelevant if this version of the agent cannot perform its defined job within its approved boundaries often enough to be useful.

That requires a test set, explicit acceptance criteria, and repeated evaluation after deployment.

Write the job as a testable contract

Begin with one sentence:

For a revenue manager, identify late-stage opportunities expected this month that have had no customer activity in 14 days, show the relevant CRM evidence, and refuse records outside the manager’s region.

This describes more than a prompt. It gives the evaluation a user, outcome, source, time window, evidence requirement, and access boundary.

“Help with pipeline” does not tell you what a correct result contains. “Answer questions about our data” makes almost every failure debatable. A narrow job lets the team distinguish product quality from model charisma.

If the team has not chosen a job yet, start with one recurring business question before designing an evaluation program.

Build a representative question set

A test set should resemble the work the agent will actually receive. Do not fill it only with happy-path questions written by the developers who built the agent.

Collect examples from request queues, review meetings, search logs, support channels, and subject-matter experts. Remove or replace sensitive data before storing tests. For every case, record:

• the user’s role and permitted scope;
• the question or triggering event;
• the expected source or tool;
• required facts and acceptable variations;
• disallowed facts or actions;
• the evidence a reviewer should see;
• whether the agent should answer, clarify, refuse, or escalate.

Include at least five kinds of cases.

Normal cases

These are common questions with sufficient context and approved data. They tell you whether the agent can do its ordinary job.

Boundary cases

Use empty periods, duplicate records, changed definitions, unusual cohorts, late-arriving data, and accounts at permission boundaries. Many misleading answers appear at the edges rather than in the average case.

Ambiguous cases

Ask questions such as “Which customers are active?” when more than one approved definition exists. The expected behavior may be to clarify or state the definition used.

Forbidden cases

Ask for another tenant, restricted fields, an unavailable tool, or an action beyond the agent’s role. A refusal can be a passing result.

Failure cases

Simulate a stale source, API timeout, malformed result, partial retrieval, rate limit, revoked permission, or unavailable dependency. The agent should not turn a system error into a confident business conclusion.

Evaluate the complete system

A model can generate a good answer while the agent system still fails. Evaluation needs to cover the path from request to result.

Task completion

Did the agent complete the job the user asked for? Check required records, comparisons, calculations, outputs, and next steps. A polished summary that omits half the requested accounts is not complete.

Factual correctness

Compare factual claims with the approved source state captured for the test. For changing data, retain a snapshot or query result so the expected answer remains reproducible.

Not every answer has one exact wording. Score the facts and relationships that must be present rather than matching the entire response as text.

Grounding and evidence

Can the reviewer connect meaningful claims to the source, period, definition, and records that support them? Check that citations actually support the claim rather than merely mentioning the topic.

This is the operational version of what trusted AI answers require: evidence is part of answer quality, not decoration added later.

Source and tool selection

Did the agent use the approved source for the question? Did it call a live query when a document index would be stale? Did it avoid unnecessary tools and stop after obtaining sufficient evidence?

A correct answer produced through the wrong access path may still indicate a design defect.

Permission and refusal behavior

Test whether the agent respects user, tenant, workspace, source, tool, and record boundaries. Confirm that a denied request stays denied when it is rephrased, embedded in retrieved content, or split across several steps.

The OWASP AI Agent Security Cheat Sheet recommends repeatable abuse cases for tool misuse, privilege escalation, data exfiltration, approval bypass, and multi-agent chaining. A security test belongs in the evaluation set, not in a separate document nobody runs.

Follow-up usefulness

Business investigation rarely ends with the first answer. Test whether follow-up questions preserve the right scope and definitions. The agent should not forget the period, switch customer segments silently, or broaden access because the user says “show me the rest.”

Operational behavior

Measure latency, tool failures, retries, token or compute use, and cost per completed job. The purpose is not to minimize every number independently. A slow but reliable weekly brief may be acceptable; a recurring workflow with unbounded retries is not.

Recovery and escalation

When the agent cannot finish, does it identify the missing source or failed step? Can it return partial evidence without presenting it as complete? Does it direct the question to the right owner?

Failure quality is part of product quality.

A production acceptance scorecard

The weights below are a starting point for an analytical agent with read-oriented access. Adjust them to match the job’s consequences. Do not hide a critical permission failure inside a high average.

Set three types of acceptance rule:

1. Overall threshold: the minimum weighted score for release.
2. Dimension floors: minimum scores for correctness, grounding, and other important areas.
3. Zero-tolerance failures: unauthorized disclosure, unapproved consequential action, fabricated evidence, or concealed source failure.

The team should choose thresholds based on the job and its risk. Copying an arbitrary passing percentage from another product makes the score look objective without making it relevant.

Use human and automated review together

Some checks are deterministic. Did the tool call succeed? Did the result include a forbidden tenant? Did the cited record contain the stated value? Did latency exceed the limit?

Other checks require judgment. Was the answer sufficiently complete? Was the clarification useful? Did the summary distinguish fact from inference? Subject-matter experts should define these rubrics and review a sample of outputs.

Model-based evaluators can help scale semantic review, but they need calibration against human decisions. Run both on the same examples, inspect disagreement, and keep a human-reviewed holdout set. Do not let one model become the unexamined judge of another.

NIST’s work on AI testing, evaluation, validation, and verification emphasizes that context is crucial and that measurement choices depend on how a system is used. See NIST’s AI TEVV program. That principle is especially important for agents: tool access, source state, user permissions, and workflow consequences are not captured by a general language benchmark.

Keep the evaluation reproducible

Record the environment behind every run:

• agent instructions and version;
• model and provider version where available;
• available tools and schemas;
• source snapshot or data version;
• semantic definitions;
• permission policy;
• test-set version;
• evaluator and rubric version;
• run time and configuration.

Without this information, a score change is hard to interpret. The model may have changed, a source view may have gained a column, or the permission layer may have rejected cases that previously leaked.

Store previously observed failures as regression tests. If a renamed CRM stage once caused the agent to omit a cohort, that case should run after later prompt, tool, or model changes.

Evaluate continuously after launch

Pre-release testing cannot reproduce every real question or source condition. Once the agent is in use, monitor:

• unanswered and escalated questions;
• low-confidence or disputed results;
• source and tool failures;
• denial and approval patterns;
• changes in task completion, latency, and cost;
• new user phrasing not represented in the test set;
• drift after data, tool, policy, prompt, or model changes.

NIST’s 2026 guidance on trustworthy AI notes that controlled pre-deployment tests cannot capture the full variation of real use and should be complemented by repeated evaluation after deployment. See NIST AI 800-4.

Treat production feedback as candidate test data, not as permission to collect unrestricted business content. Preserve the minimum evidence needed, apply retention rules, and involve the source owner when a new failure reveals a definition or data-quality problem.

A practical release process

1. Define the job and zero-tolerance failures.
2. Assemble representative, boundary, forbidden, and failure cases.
3. Capture expected facts, sources, actions, and refusal behavior.
4. Run deterministic checks and human-scored rubrics.
5. Review failures by category rather than chasing only the average.
6. Fix the source, definition, tool, permission, instruction, or model at the layer that caused the failure.
7. Rerun the full set and compare versions.
8. Release to a bounded group with monitoring and an escalation owner.
9. Add real failures to the regression set.

An evaluation program should make the agent’s limits more visible, not create a single number for a slide.

Jovis lets teams create agents within workspaces that contain approved business context and inspect grounded answers. Before expanding an agent to more users or sources, turn the recurring questions it is meant to answer into a versioned test set. That is the fastest way to move from an impressive demo to evidence that the workflow is ready.