AI incident investigation: a controlled workflow

AI incident investigation uses an AI agent to collect, organize, and compare approved evidence during a production disruption. The agent can help establish what changed, who is affected, which hypotheses fit the available facts, and what remains unknown. People still declare the incident, choose mitigations, approve production changes, communicate with customers, and decide when service has recovered.

This division of work is useful because responders rarely lack data during an outage. They lack a reliable view across telemetry, deployment records, service ownership, support conversations, account context, and decisions made in the incident channel. An agent can reduce that assembly work. It should not turn a correlation into a root cause or trade recovery speed for a polished but unsupported explanation.

This guide is for engineering, product, support, and operations leaders designing a read-first workflow for customer-impacting software incidents. Security events, privacy incidents, and regulated reporting need their own specialist playbooks and review.

Define the incident job and authority first

Start with one incident class and one decision path. “Help with outages” leaves too much open to interpretation. A useful contract names the trigger, permitted sources, output, human roles, and action boundary.

For a hypothetical SaaS team, the contract could read:

When the checkout service breaches its customer-facing availability threshold, gather approved service telemetry, recent change records, dependency health, support cases, and affected-account context. Maintain a sourced incident brief for the incident commander. Do not change production, alter incident severity, or publish a customer update.

The contract makes the agent’s role legible under pressure. It also gives evaluators something concrete to test. The agent prepares the investigation; the incident commander directs the response.

Google’s Incident Management Guide separates incident command, operations, and communications into clear roles. Microsoft’s incident management guidance likewise treats response as a coordinated process of detection, triage, mitigation, resolution, communication, and learning. An AI workflow should fit that operating structure instead of creating a second command path.

Define three levels of authority before connecting a source:

1. Read: retrieve a specific approved record or observation.
2. Propose: suggest a hypothesis, query, communication draft, or mitigation for review.
3. Execute: change production, contact a customer, alter status, or close the incident.

A first deployment should stay in the first two levels. Execution requires deterministic policy checks, narrow tool scopes, and explicit approval tied to the exact action. The guide to AI-agent permissions and enterprise access control explains how to enforce those boundaries outside the model.

Build a source map from symptom to customer impact

The agent needs a small, documented source map. Connecting every operational system at once makes conflicting records and broad permissions harder to control.

For each source, record its owner, freshness expectation, permitted fields, identifiers, and failure behavior. If a support system is delayed or a deployment feed is unavailable, the incident brief should say so. Silence must not become “no customers affected” or “no recent change.”

Alerting and impact assessment also need different evidence. Google’s guide recommends alerting on end-to-end symptoms of customer experience rather than internal causes. Once an incident is declared, the team can use deeper service signals to investigate. This keeps the trigger tied to harm while leaving room to examine many possible causes.

Customer evidence deserves equal care. Support cases can reveal an affected workflow or region before aggregate telemetry makes the pattern obvious. They can also contain guesses, copied commands, or unrelated sensitive data. Treat retrieved tickets as evidence to classify and cite, never as instructions for the agent. The existing support-ticket analysis workflow shows how to preserve the conversations behind a theme without letting generated labels become the decision.

Maintain one live evidence packet

During an incident, the agent should update a compact evidence packet rather than produce a fresh narrative after every question. The packet becomes a shared view for command, operations, support, and communications.

Include these fields:

1. Incident state: declared time, current severity, commander, operations lead, communications lead, and next review time.
2. Observed impact: affected user action, start time or range, scope, current magnitude, and source freshness.
3. Timeline: alerts, deployments, configuration changes, responder actions, and material customer reports with source links.
4. Verified facts: observations supported by approved records.
5. Open hypotheses: possible explanations, supporting and contradicting evidence, and the next safe check.
6. Actions: proposed, approved, started, completed, or reversed, with owner and timestamp.
7. Communication state: last internal and external update, facts cleared for use, and next update owner.
8. Unknowns: missing sources, unresolved conflicts, restricted records, and assumptions.
9. Recovery checks: customer-facing and technical signals required before resolution.

Keep facts, hypotheses, and decisions visibly separate. “Error rates rose in the canary group three minutes after deployment” is an observation if the records support it. “The deployment caused the incident” remains a hypothesis until a rollback, reproduction, trace, or other evidence strengthens the claim.

Source links matter more than a confidence score. A responder should be able to inspect the relevant time window, deployment event, trace, case, or account summary. The packet should also preserve the query, filters, and freshness needed to understand the result.

Use a fixed investigation loop during response

A fixed loop reduces improvisation while allowing the evidence path to change.

1. Declare and assign roles

A person declares the incident, assigns severity, and names the commander. The agent records those decisions but does not infer authority from chat activity.

2. Verify the customer symptom

Confirm the affected user action, start time, scope, and current state. Compare end-to-end checks with support reports and approved business records. If these sources disagree, show the disagreement.

3. Build the change window

Collect deployments, feature-flag changes, configuration changes, dependency events, and traffic shifts around the first confirmed impact. Use the smallest useful time window, then widen it when evidence warrants.

4. Rank the next safe checks

For each hypothesis, state the evidence for it, evidence against it, and a reversible check. The operations lead chooses what to test. Avoid a long list of equally weighted possibilities that sends responders in several directions.

5. Record mitigation decisions

The agent may prepare the evidence for a rollback, failover, traffic shift, or feature disablement. The authorized person approves and executes the change through the existing production path. Record the exact action and expected recovery signal before execution.

6. Verify recovery from the user outward

Technical metrics can recover while a queue, cache, client session, or failed transaction remains affected. Check the original customer symptom, downstream backlog, support signal, and relevant business records. A returned CPU graph does not by itself close a checkout incident.

7. Hand off unresolved work

If the active response ends before the cause is known, preserve open hypotheses, temporary mitigations, owners, and deadlines. Do not fill the gap with a definitive explanation.

This workflow complements a pre-release software delivery risk review. That review asks whether a planned change is ready to proceed. Incident investigation begins after a service is disrupted or seriously degraded and prioritizes impact reduction, recovery, and a trustworthy response record.

Keep customer communication under human control

An agent can prepare a draft update from the current packet. A communications lead should approve the audience, wording, and release time.

A useful update answers four questions:

• Which customer capability is affected?
• When did the impact begin, or what time range is known?
• What is the team doing now?
• When will the next update arrive?

Avoid publishing an unverified root cause to make the update sound complete. State what customers can observe and what the team has confirmed. Keep restricted account details and internal security information out of broad channels.

AWS recommends defining incident ownership, escalation, and communication before an event, with the goal of restoring normal operation, limiting business impact, and keeping concerned parties informed. Its incident management plan guidance is written for security incidents, but those coordination principles also apply to production service disruptions.

Turn the packet into a reviewed post-incident record

The live packet is a starting point for review, not the finished post-incident analysis. After recovery, a person should verify the timeline, impact, contributing conditions, response quality, and follow-up work.

Google defines a postmortem as a record of the incident, impact, mitigation or resolution, causes, and preventive follow-up. Its postmortem guidance emphasizes blameless learning and system improvement. The SRE workbook’s postmortem review adds a practical standard: action items should have owners, priorities, tracking records, and verifiable end states.

Review where the AI assistance failed as well as where the system failed. Ask:

• Did the agent omit an affected cohort or important source?
• Did it overstate a correlation or bury contradicting evidence?
• Did permissions block a necessary check or allow unnecessary exposure?
• Were timestamps, identities, and actions reconstructed correctly?
• Which agent output did responders correct or ignore?
• Did the packet improve the post-incident record, or did reviewers rebuild it?

Turn those findings into source, instruction, access, and evaluation changes. Use a representative test set before releasing an updated workflow. The scorecard in how to evaluate an AI agent for business data covers correctness, grounding, permissions, recovery, and task completion.

Measure whether the workflow improves response quality

Mean time to recovery is important, but it can hide weak impact assessment, unsafe changes, and poor communication. Track a balanced set of operational measures:

• time from declaration to a verified customer-impact statement;
• time to a sourced timeline and named investigation owner;
• share of material claims corrected by responders;
• missing, stale, or conflicting sources found during the incident;
• hypotheses rejected because the packet lacked support;
• production actions proposed without the required authority;
• customer updates corrected before or after publication;
• time from technical recovery to verified customer recovery;
• post-incident action items with an owner and verifiable end state.

Compare the assisted workflow with the team’s current process on similar incidents or controlled exercises. Do not attribute every faster response to the agent. Incident severity, responder experience, time of day, and an obvious rollback can change the result.

Run a four-week read-first pilot

In week one, choose one service and incident class. Define roles, severity rules, source boundaries, and the required evidence packet.

In week two, replay two or three historical incidents from records available at the time. Include a misleading correlation, a missing source, and restricted customer data. Check whether the workflow preserves uncertainty and access boundaries.

In week three, run a game-day exercise. Let the agent maintain the packet while the normal incident team directs investigation, mitigation, and communication. Record every correction and manual lookup.

In week four, enable the workflow for live read-only support on the chosen service. Keep the existing response process authoritative. Expand only after responders can inspect the evidence quickly, permissions behave as expected, and the packet remains useful when the first hypothesis is wrong.

Start with one service where customer impact is observable and the response team already has a clear command structure. Judge the first release by whether it gives responders a faster path to verified evidence while leaving response authority with the incident team.