An agent that finishes in one HTTP request can live in application code. An agent that waits for approval, retries a vendor, or resumes tomorrow needs a durable state machine.
The distinction is easy to miss. The model call succeeds in a demo, so the team adds a queue and calls the system reliable. Then a worker crashes after sending an email but before acknowledging the task. The retry sends the email again.
The model isn't the hard part. The recovery boundary is.
Separate decisions from effects
Treat the workflow as two kinds of operation:
| Operation | Example | Required property |
|---|---|---|
| Decision | Choose the next tool | Replayable or recorded |
| Effect | Charge card, send email, write CRM | Idempotent and observable |
| Wait | Human approval, timer, webhook | Durable and addressable |
| State transition | draft → approved | Atomic and validated |
Model inference is a decision with an external dependency. You can either record its result and replay from that result, or call it again and accept that the answer may change. For most business workflows, recording the result is safer.
Effects need stable idempotency keys. The retry boundary described in Idempotency and Retries for Reliable AI Agents still applies when a workflow engine handles scheduling.
idempotency_key = workflow_id + step_name + logical_attempt
Do not use a random key on every retry. That turns duplicate protection off precisely when you need it.
Model the workflow explicitly
A useful agent state is boring and inspectable:
{"workflow_id": "wf_82J","status": "awaiting_approval","step": "approve_refund","input_version": 3,"decision_ids": ["dec_1", "dec_2"],"pending_effect": null,"wake_at": null,"deadline": "2026-08-23T17:00:00Z"}
Persist business state, not a serialized process with open sockets and in-memory clients. A different worker should be able to continue from the record.
The workflow should also reject impossible transitions. paid → draft is not an innocent retry. It is a state corruption bug.
Design every step for interruption
For each step, ask four questions:
- What durable fact proves the step started?
- What durable fact proves it completed?
- What happens if the worker stops between those writes?
- Can the effect be checked before it is repeated?
The dangerous gap is usually between the external effect and the completion record:
call payment API → worker crashes → save completed
Close it with a provider idempotency key, an outbox, or a reconciliation query. A workflow engine can retry the step; it cannot make an unsafe API idempotent on your behalf.
Treat human approval as an event
An approval is not a worker sleeping in a loop. Store a waiting state, expose an authenticated event endpoint, and correlate the event with the workflow and expected state.
Validate all three:
- the approver has permission;
- the workflow is still awaiting this approval;
- the event has not already been consumed.
Late approvals are normal. If the deadline passed or the request was superseded, return the current state rather than resurrecting the old branch. See Human-in-the-Loop Patterns for AI Agents for approval boundaries and escalation rules.
Choose the recovery mechanism deliberately
| Need | Reasonable mechanism |
|---|---|
| Minutes, few steps, no human wait | Database job plus idempotent worker |
| Hours or days, timers and signals | Durable workflow engine |
| High event volume, simple transitions | Event stream plus state projector |
| Regulated business process | Workflow history plus append-only audit log |
Temporal records workflow event history and reconstructs workflow state through replay. Cloud queues provide durable delivery, but the application still owns multi-step state, timers, and effect safety. Those are different guarantees.
Test recovery, not only success
Kill workers at each boundary in a staging environment:
- before and after every external effect;
- while waiting for an approval;
- after receiving the same webhook twice;
- after a timeout whose remote result is unknown;
- during a deployment with an in-flight workflow;
- after the workflow definition changes.
The pass condition is not “the workflow eventually completed.” Check that no effect happened twice, the audit history explains every transition, and an operator can repair the workflow without editing raw rows.
Durable-agent readiness checklist
- Every workflow has a durable identifier and explicit state.
- Decisions and external effects are recorded separately.
- External effects use stable idempotency keys or reconciliation.
- Waiting does not occupy a worker.
- Signals and webhooks are authenticated and deduplicated.
- Workflow code changes account for in-flight executions.
- Operators can inspect, retry, cancel, and repair a workflow.
- Failure-injection tests cover every effect boundary.
