Idempotency and Retries for Reliable AI Agents

Retries make AI agents more reliable until the retried operation sends the same email twice, creates two support tickets, or charges a customer again.

The model is not the core problem. Distributed systems cannot always tell whether an operation failed before or after the side effect occurred. Agents amplify that ambiguity because they can retry at several layers: the HTTP client, model provider, agent loop, queue, workflow engine, or a human resume action.

Reliable agent architecture therefore begins with one rule:

Any operation that may run more than once must either be idempotent or have an explicit duplicate-prevention strategy.

This guide extends the production agent design patterns into the failure semantics of real tool execution.

Why “exactly once” is usually the wrong mental model

Imagine an agent calling a payment API:

1. The payment provider creates the charge.
2. The network connection drops before the agent receives the response.
3. The agent sees a timeout.
4. A retry creates a second charge.

From the agent's perspective, the first call failed. From the external system's perspective, it succeeded.

Common execution semantics are:

• At-most-once: do not automatically repeat an uncertain action; it may never happen.
• At-least-once: retry until completion; the action may happen more than once.
• Effectively-once: allow retries but deduplicate the resulting effect with a stable identity.

Most reliable workflows use at-least-once delivery plus idempotent operations to approximate effectively-once business behavior.

Classify tools by side effect

Do not apply one retry policy to every tool.

The classification belongs in tool metadata, not in a prompt. The runtime should know whether an operation is safe to retry even if the model does not.

Generate idempotency keys from business intent

An idempotency key identifies one intended business operation across attempts.

Python

operation_key = f"refund:{order_id}:{approved_refund_request_id}"

Every retry of that refund uses the same key. A genuinely new refund receives a new request ID.

Avoid keys based only on timestamps or random values generated inside each attempt; those make every retry look unique. Avoid hashing the full natural-language request as the sole identity, because harmless wording changes can produce a second side effect.

Store an operation record before or atomically with execution:

operation_key
tool_name
validated_arguments
status: pending | succeeded | failed | uncertain
external_receipt_id
result
created_at
updated_at

When a duplicate request arrives, return the stored result or continue reconciliation rather than running the side effect again.

Treat “uncertain” as a real state

Binary success/failure status is insufficient. A timeout after submission is not a confirmed failure.

Use an explicit uncertain state:

pending → succeeded
pending → failed
pending → uncertain → succeeded
pending → uncertain → failed

The recovery worker can query the external system using the idempotency key or receipt. If the external API offers no lookup mechanism, route high-impact uncertainty to human review rather than guessing.

This also improves observability. A timeout counter says the network was slow. An uncertain-operation counter says the business may be inconsistent.

Put retries in one layer when possible

Nested retries multiply unexpectedly. Three attempts in the HTTP client, three in the tool wrapper, and three in the workflow engine can produce 27 calls.

Choose one layer to own each retry class:

• transport client: transient connection establishment failures;
• tool adapter: provider-specific throttling or temporary errors;
• workflow engine: durable retries across process restarts;
• agent policy: reconsidering the plan after a semantic tool failure.

Expose attempt counts and the final error between layers. Do not let an inner retry loop hide ten seconds of latency and six paid calls behind one tool span.

Retry only errors that may improve

Useful retry categories:

• rate limits with a server-provided retry delay;
• transient network failures;
• temporary provider unavailability;
• optimistic-concurrency conflicts when the operation can be recomputed safely.

Do not automatically retry:

• invalid arguments;
• authentication or authorization failures;
• policy denials;
• missing required user confirmation;
• deterministic schema errors;
• insufficient funds or other business-rule failures.

Use exponential backoff with jitter for transient errors. Cap attempts and total elapsed time. A dead-letter or review queue is better than an immortal retry loop.

Separate model retries from tool retries

A model retry asks the model to produce a new output. A tool retry repeats an external operation. They have different risks.

If a tool returns an argument-validation error, the model may correct the arguments and create a new attempt. If a tool times out after an uncertain write, asking the model to “try again” can duplicate the effect.

Return structured failure information to the orchestrator:

{
  "status": "uncertain",
  "retryable": false,
  "operation_key": "ticket:case_812:initial_escalation",
  "next_action": "reconcile"
}

The model can explain the delay to the user, but it should not override the runtime's retry decision.

Design compensation for multi-step workflows

Some operations cannot be made idempotent as a group. Consider:

reserve inventory → charge card → create shipment

If shipment creation fails, repeating the entire workflow is wrong. Record completed steps and define compensating actions such as releasing inventory or refunding the charge.

Compensation is not a database rollback. External actions may be delayed, rejected, or only partially reversible. Treat compensation as another observable workflow with its own idempotency keys and escalation path.

Test failure windows deliberately

Happy-path tests do not validate retry safety. Inject failures:

• immediately before the external call;
• after submission but before the response;
• after the response but before saving the receipt;
• during state persistence;
• after process termination and workflow replay;
• while two workers execute the same operation concurrently.

Assert the final business state, not only the number of successful function returns. The key test is that one intended operation creates no more than one business effect.

Connect these tests to the trace schema in LLM observability, including operation key, attempt number, retry owner, result state, and external receipt.

Reliability checklist for agent tools

• Every tool declares its side-effect and retry class.
• Business operations receive stable idempotency keys.
• Duplicate attempts return stored results or reconcile state.
• Uncertain outcomes are distinct from confirmed failures.
• One layer owns each retry policy.
• Non-retryable errors are represented explicitly.
• Attempts and total elapsed time are capped.
• Multi-step workflows define compensation and escalation.
• Failure-injection tests cover every persistence boundary.
• Traces record attempts, operation keys, and receipts.

Sources and further reading

• AWS Durable Execution: idempotency and retries
• OpenAI Agents SDK: durable execution integrations
• Temporal durable execution technical guide