An AI agent system design interview should test whether you can bound model-driven decisions inside a reliable software system. The difficult part is not drawing an LLM connected to tools. It is defining which decisions may be probabilistic, which controls must remain deterministic, and what happens when a tool, model, user, or dependency behaves unexpectedly.
These questions are independent practice material, not a description of any employer's interview loop. Confirm the actual format and allowed tools with the recruiter.
Use one design sequence
Move through the same sequence for every prompt:
- Task: user, desired outcome, non-goals, and cost of a wrong action.
- Authority: identity, permissions, approvals, and irreversible boundaries.
- Tools: narrow contracts, validation, idempotency, timeouts, and error classes.
- State: durable task state, conversation context, provenance, and retention.
- Control flow: deterministic workflow, single-agent loop, routing, or multi-agent coordination.
- Evaluation: representative tasks, trajectory checks, outcome checks, and release gates.
- Operations: latency/cost budget, traces, fallback, recovery, and incident owner.
Do not start by selecting an agent framework. Architecture follows the task and authority boundary.
Scoring rubric
Score each dimension from 0 to 2:
| Dimension | 0 | 1 | 2 |
|---|---|---|---|
| Task contract | Vague assistant | Names a workflow | Defines success, refusal, and failure cost |
| Authority | Model decides freely | Mentions approval | Locates identity, policy, and irreversible gates |
| Tool design | Lists integrations | Defines schemas | Adds validation, idempotency, error, and retry rules |
| State/recovery | Keeps chat history | Persists task state | Separates state types and designs replay/recovery |
| Evaluation | Says “test it” | Names metrics | Defines cases, trajectory/outcome checks, and thresholds |
| Operations | Adds logging | Names latency/cost | Defines trace, objectives, fallback, rollback, and ownership |
A strong response scores 10–12 and survives follow-up changes without moving critical authorization into the prompt.
Core interview questions
1. When should a workflow be an agent?
Use an agent when the task needs model-driven decisions about which step or tool to use based on changing intermediate results. Prefer deterministic code when the sequence and rules are known. OpenAI's agent guide recommends agents for workflows with complex decisions, difficult-to-maintain rules, or substantial unstructured input; Microsoft's architecture guidance similarly distinguishes fixed chains from adaptive agent behavior.
Follow-up: what evidence would justify adding a second agent?
2. Design a tool contract for issuing a refund
Separate the model-visible proposal from the server-authorized execution. The tool should accept typed fields, derive customer and operator identity from trusted context, validate amount and reason, enforce policy and approval thresholds, use an idempotency key, and return a stable result or classified error. The model must not supply its own authorization scope.
Follow-up: how do you prevent a retry from issuing two refunds?
3. Where should human approval occur?
Place approval immediately before the side effect, after the exact action and parameters are known. Bind approval to that proposal and expire it when parameters change. High-impact, irreversible, legally sensitive, or low-confidence actions should pause or escalate.
Follow-up: what information does the reviewer need, and what should be hidden?
4. How do you store agent state?
Distinguish conversational context, durable workflow state, tool results, long-term user memory, and audit evidence. Give each a schema, owner, retention rule, and access boundary. A task state machine should survive process restarts without depending on an opaque transcript.
5. How do retries work in an agent loop?
Retry only classified transient failures, use bounded attempts and backoff, and keep a task-level budget. Never blindly retry a side effect. Use idempotency or a read-before-reconcile path. Store the attempt and tool result so recovery does not rerun successful work.
6. Single agent or multiple agents?
Start with one bounded agent plus tools. Add specialized agents only when distinct domains, context limits, security boundaries, parallelism, or independently evaluated responsibilities justify coordination overhead. Microsoft notes that multi-agent systems require both component-level and end-to-end monitoring and testing.
7. How do you stop runaway execution?
Set maximum steps, tool calls, elapsed time, tokens, and spend per task. Detect repeated calls and no-progress states. Make cancellation observable, propagate deadlines to tools, and return a safe partial result or escalation rather than looping.
8. What belongs in an agent trace?
Record sanitized task and actor identifiers, policy/model/prompt/tool versions, state transitions, tool names and validated arguments, results or error classes, approvals, token/latency/cost fields, final outcome, and evaluation labels. Avoid raw secrets and unnecessary personal data.
9. How do you test tool selection?
Build cases for correct selection, correct non-selection, invalid arguments, conflicting tools, tool failure, prompt injection in tool output, and permission boundaries. Score both the final outcome and the trajectory; a correct answer reached through an unauthorized call is still a failure.
10. How do you handle prompt injection from retrieved content?
Treat retrieved text and tool output as untrusted data. Keep policy and authority outside retrieved content, delimit sources, constrain available tools, validate calls server-side, and require approval for risky actions. Add adversarial cases that attempt to override instructions or exfiltrate data.
11. How do you release a new model or prompt?
Replay a versioned offline set, compare outcome and trajectory slices, shadow or canary the change, enforce critical-boundary gates, and retain a rollback pointer. Monitor cost and latency alongside task success. Provider release notes are not a substitute for application evaluation.
12. How does the system degrade safely?
Define behavior for model timeout, tool outage, exhausted budget, stale data, evaluation drift, and approval unavailability. Options include read-only mode, deterministic search, a draft without execution, queueing for review, or a clear refusal.
Complete design exercise
Prompt: Design an agent that triages customer support email, looks up account and order data, drafts a response, and may issue refunds up to a policy limit.
Requirements to establish
- Is the agent drafting, sending, or both?
- Which inboxes, regions, and languages are in scope?
- What counts as successful resolution?
- Which refunds require approval?
- How quickly must the first response arrive?
- What data may be stored and for how long?
A defensible architecture
email event-> identity + tenant scope-> deterministic classification / policy precheck-> bounded agent looptools: read_customer, read_order, search_policy, propose_refund-> output and citation validator-> approval gate for send/refund when required-> idempotent executor-> audit + outcome event
The propose_refund tool should not move money. It returns a typed proposal with policy evidence. A deterministic service checks authorization and approval before calling the payment provider. Email content cannot change those rules.
Evaluation set
Include routine questions, ambiguous identity, missing orders, conflicting policy, duplicate messages, already-refunded orders, prompt injection, tool outage, high-value refunds, and multilingual messages. Score:
- correct disposition and response;
- policy citation and groundedness;
- tool/argument correctness;
- prohibited-action rate;
- duplicate-side-effect rate;
- escalation quality;
- latency and cost per resolved case.
Set critical gates independently of the average: zero unauthorized refunds and zero cross-customer disclosure in the release set.
Follow-up changes
Practice adapting the design when:
- refund policy changes daily;
- one region requires human approval for every refund;
- the payment API times out after accepting a request;
- a second specialist agent is proposed for billing;
- p95 latency must be cut without weakening approval.
Practice method
Spend five minutes clarifying, twenty-five minutes designing, ten minutes on failure and evaluation, and five minutes summarizing tradeoffs. Record which claims lacked an artifact or test. Then compare your answer with the production AI agent architecture guide and the single-agent versus multi-agent decision framework.
For broader preparation, use the four-week AI engineer interview plan and complete one exercise from the JoinAI problem catalog under time constraints.
