LLM Incident Response Runbook

An LLM incident is any model, prompt, retrieval, tool, policy, or provider failure that materially harms users, the business, or system security. Respond as an operational incident: assign command, contain impact, preserve evidence, recover through an explicit gate, and convert the failure into a regression test.

The key difference from ordinary uptime response is that an LLM endpoint can return 200 OK while producing unsafe, ungrounded, or unauthorized results.

Severity matrix

Adapt these examples to your product and obligations before an incident occurs.

Severity follows impact, not technical novelty. One cross-tenant disclosure can be more severe than thousands of harmless formatting failures.

First 15 minutes

Use this as the copyable opening checklist:

[ ] Assign incident commander, technical lead, and communications owner
[ ] State observed impact, affected users, and current severity
[ ] Freeze unrelated prompt, model, index, policy, and tool changes
[ ] Identify the first known bad time and last known good version
[ ] Contain: kill switch, rollback, route, restrict tools, or safe fallback
[ ] Preserve trace IDs and version identifiers under the retention policy
[ ] Start a timestamped decision log and set the next update time

Containment outranks root-cause certainty. If a write-capable agent may be acting outside authorization, remove the capability before debating whether the prompt or model caused it.

Classify the incident by impact surface

This classification organizes the response; it is not yet a root cause. Use the AI agent failure taxonomy to walk backward to the earliest broken contract.

Preserve evidence without creating a second incident

Record stable identifiers for the trace, model, prompt, tool set, policy, retrieval index, deployment, tenant, and evaluator. Preserve tool receipts and the timeline of state transitions.

Do not indiscriminately copy raw prompts, credentials, customer documents, or regulated data into an incident channel. Follow the existing capture, access, and retention policy. Redaction should preserve the evidence needed to reproduce the boundary.

If compromise or prompt injection is plausible, rotate exposed credentials and isolate the execution path. Do not replay a suspected attack against production. Rebuild the trust boundaries in a controlled environment and apply the prompt-injection regression plan.

Choose the smallest safe containment

Containment options, from narrow to broad, include:

• block one tenant, intent, tool, document source, or prompt version;
• make a write tool read-only or require human approval;
• route to a previously qualified model or prompt;
• disable retrieval from a compromised index;
• return a deterministic answer or queue the task for humans;
• activate the feature kill switch.

A model swap is not automatically a safe rollback. The previous route must still be compatible with the current prompt, tool schema, and policy. For uncertain side effects, preserve one operation identity and reconcile the provider receipt before retrying; the idempotency guide covers that boundary.

Recovery gate

Do not recover because five hand-picked prompts look correct. Require evidence:

1. The incident trigger is represented in a regression case.
2. The proposed repair passes deterministic policy and authorization checks.
3. The relevant quality slice passes its release threshold.
4. Tool side effects, timeout, rollback, and fallback paths are exercised.
5. A canary shows normal impact, cost, and tail latency.
6. The commander records the recovery decision and monitoring window.

Build the permanent regression into the LLM evaluation dataset, including the expected safe refusal, clarification, or escalation when that is the correct outcome.

Incident record template

Impact:
Severity and rationale:
First known bad / last known good:
Affected versions and traffic slices:
Containment and timestamp:
Earliest broken contract:
Root cause and contributing conditions:
Recovery evidence:
Corrective actions, owners, and due dates:
Regression test:

The postmortem should be blameless and specific. “The model hallucinated” is not actionable. “The release gate had no groundedness test for policy questions, and the new index omitted the governing document” identifies contracts and owners.

NIST SP 800-61 Rev. 3 frames incident response as part of broader risk management, including preparation, detection, response, recovery, and continuous improvement. Google SRE’s guidance similarly emphasizes a written record, ownership, and tracked action items. Apply those disciplines to quality and agent behavior, not only cybersecurity and uptime.

Prepare before the page

• Severity definitions are approved by security, product, and operations.
• Every production AI feature has a tested kill switch and owner.
• Trace evidence includes prompt, model, index, policy, and tool versions.
• Security and privacy rules govern incident evidence.
• Fallbacks are tested under provider and tool failure.
• Communication templates and customer escalation paths exist.
• Corrective actions feed release evals and operational alerts.

Pair this runbook with the production LLM observability schema. If your team needs to rehearse the workflow against its own architecture, JoinAI can help design an incident simulation and readiness plan.

Sources and further reading

• NIST SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management
• Google SRE Workbook: Postmortem culture
• OpenTelemetry: Generative AI semantic conventions registry