Data quality for AI agents: a practical control plan

Data quality for AI agents is the fitness of a specific source snapshot for the job an agent is about to perform. A useful control plan tests that fitness before the agent uses the data, defines what happens when a check fails, and preserves enough evidence for an owner to diagnose the problem.

Define quality at the job boundary instead of trying to “clean all enterprise data.” A weekly pipeline agent can ignore many optional CRM fields. It still needs the correct opportunity population, current stage and close-date fields, reliable account identifiers, the approved pipeline definition, and a clear response when one of those inputs is not fit for the forecast decision.

The practical standard is therefore fit for this job, at this time, for this user and consequence. The Australian Government’s current agentic AI data guidance takes a similarly risk-based position for agencies: data readiness, governance, and security should be confirmed as fit for the system’s level of autonomy before progressing beyond early experimentation.

For a business-data team, that principle becomes an executable contract rather than a general aspiration.

Start with the decision that bad data could distort

A generic quality score cannot tell you whether an agent run is safe to use. Ten optional CRM fields may be empty while the three fields required for today’s decision are complete. Conversely, a table may pass 99 percent of its rules while one broken currency field makes a cash forecast unusable.

Write the agent’s job in one sentence before designing checks:

Every Monday, identify late-stage opportunities expected to close this quarter that need manager review, using the approved CRM snapshot and activity records. Show the evidence for each item, and do not produce a complete regional brief if the required snapshot, ownership scope, or opportunity identifiers are invalid.

That statement reveals the protected decision, cadence, population, sources, evidence, and stop condition. It also narrows the quality surface. The team can now ask which failures would change the population, misstate a material fact, hide an affected record, or expose data outside the manager’s scope.

If the source pattern is still undecided, first compare the tradeoffs in connecting AI agents to enterprise data. Live queries, synchronized tables, document indexes, and governed APIs fail in different ways and need different checks.

Define six dimensions of fitness

Most agent workflows need checks across six dimensions. The exact rules and thresholds should come from the job, not from a universal checklist.

The first five dimensions are often testable with deterministic queries, schema checks, counts, timestamps, referential-integrity rules, and reconciliations. Semantic consistency requires an approved definition and effective version. The implementation guide to a semantic layer for AI agents explains how to encode grain, entities, joins, measures, time rules, and vocabulary so the agent does not have to guess them.

Do not collapse these dimensions into “accuracy.” A value can be correctly copied from its source and still be stale, out of scope, or wrong for the business definition. Keeping the failure classes separate makes the response easier to choose and the owner easier to identify.

Write a quality contract the workflow can enforce

For each critical input, create a small, versioned contract. It should be understandable to the business owner and executable by the data platform.

Quality platforms can execute many of these mechanics. Google Cloud’s current data-quality scan documentation describes rules for completeness, validity, consistency, uniqueness, and custom SQL, as well as scheduled scans and queries for finding failed records. AWS Glue similarly supports explicit rules, anomaly detection, and identifying or quarantining failed records.

Those tools do not decide whether a failure should stop your agent. That decision belongs in the workflow contract.

Choose block, degrade, warn, or quarantine deliberately

Every failed check needs a predetermined response. Sending all failures to a dashboard leaves the agent free to produce a confident answer while someone else notices the problem later.

Block the run

Block when the failure makes the requested answer materially unreliable or unauthorized. Missing regional partitions, an expired financial snapshot, broken tenant scope, or an invalid join that duplicates revenue are typical blockers.

Return a useful failure state: which source failed, which protected claim is unavailable, when the last valid run occurred, and who owns the next step. Do not replace the blocked result with model-generated estimates unless estimation is an explicitly approved part of the job.

Degrade the result

Continue with a narrower answer when the remaining evidence can still support a clearly bounded result. If support tickets are temporarily unavailable, a customer-health workflow might report product-usage changes while stating that the review is incomplete and withholding any conclusion that depends on support context.

The reduced scope must be visible in the output and trace. “Completed” and “completed without one required source” are different states.

Warn the reviewer

Warn when the issue does not invalidate the answer but could affect interpretation. A small number of unmatched account IDs below an approved tolerance may be acceptable for a low-consequence internal review if the omitted records and their likely impact are shown.

Warnings need an expiry or review rule. A permanent yellow banner becomes decoration.

Quarantine records

Isolate individual records when the valid population can proceed without them and exclusion will not create a misleading result. Preserve the reason, original value, rule version, and repair status. Never let an agent silently “fix” an identifier, date, category, or financial value because the correction looks plausible.

An AI system may help cluster failures, compare records, or propose a mapping. A deterministic rule or accountable steward should approve material corrections before those values become source truth.

Put gates at the consumption boundary

Warehouse tests cover only part of the agent’s data path. The data may pass a pipeline check at 2 a.m. and become stale by the 9 a.m. review. A source may be healthy overall while the requesting user’s permitted cohort is empty. A retrieval tool may truncate results even though the underlying table is complete.

Run checks at three points:

1. Before access: Confirm source availability, approved version, freshness, schema, and authorization for the job and user.
2. During assembly: Validate pagination, row counts, joins, duplicate handling, document versions, and coverage of the requested cohort.
3. Before release: Confirm that required sources and evidence are present, limitations are disclosed, and the completion state matches what the agent actually obtained.

This control path should emit a stable run ID and retain rule results alongside source and configuration versions. The broader guide to AI agent observability shows how to connect those events to tool calls, permission decisions, output quality, and business outcomes.

Use anomaly detection as a signal, not a verdict

Fixed rules work well for known contracts: non-null identifiers, allowed values, reconciliation totals, required partitions, and maximum source age. They will not catch every novel failure. A sudden row-count drop, distribution shift, or unusual null pattern may reveal a problem the team did not anticipate.

Use statistical or machine-learning detection to propose an anomaly for review. Then ask:

• Did a business event legitimately change the population?
• Did a source or transformation change?
• Is the comparison baseline appropriate for seasonality and cadence?
• Which records account for the movement?
• Should this observed failure become a deterministic rule?

Avoid allowing an anomaly score alone to block a consequential workflow. The detector may be wrong, and the team needs an explainable route to override it. Conversely, do not let a high aggregate quality score override one failed critical rule. Severity comes from the decision a rule protects, not from averaging unrelated checks.

Give each failure one accountable route

Data-quality work stalls when every alert goes to “the data team.” Assign ownership by failure layer:

• a source owner handles extraction availability and schema changes;
• a data-product owner handles transformations, joins, and published contracts;
• a business owner approves definitions, tolerances, and acceptable exclusions;
• a workflow owner decides whether the agent blocks, degrades, or resumes;
• a security or privacy owner reviews failures involving restricted data or access scope.

One person may hold several roles in a small team, but the response still needs a named destination. Record acknowledgement, repair, waiver, and resumption as distinct events. A waiver should identify who approved it, which runs it covers, and when it expires.

This reflects a broader risk-management principle in the NIST AI Risk Management Framework Core: data suitability and system knowledge limits should be documented, human oversight should be defined, and systems should be able to fail safely when operating beyond those limits.

Test the gate with realistic broken data

Do not launch after proving that clean data passes. Build a failure set around the actual job:

• delay one required source beyond its freshness limit;
• remove a region or customer segment;
• introduce a duplicate join key;
• add an unknown enum value;
• change a metric definition without updating the workflow version;
• return a partial page from an API;
• make one source unavailable to the requesting role;
• create a conflict between two authoritative-looking records.

For each case, verify the rule result, response, user-facing limitation, owner notification, repair evidence, and recovery path. The AI agent evaluation guide can carry these cases into a repeatable release set alongside correctness, grounding, permissions, and task-completion checks.

Track operating measures that reveal whether the control plan helps:

• critical failures detected before an answer was released;
• false blocks and inappropriate overrides;
• time from detection to an accountable owner;
• runs degraded or quarantined by source and rule;
• repeated failures without a permanent fix;
• answers corrected because a quality check was missing;
• agent runs that claimed completeness after partial retrieval.

A predictable boundary matters more than a perfect quality score. The agent should use data that is fit for its job, stop when a critical assumption breaks, and tell the right person what needs repair.

Jovis gives teams a governed workspace for agents operating on approved business context. To evaluate it, choose one recurring job and bring the failure set above. Confirm that stale, incomplete, conflicting, and unauthorized inputs produce the response your team specified before you expand the workflow.