How to Evaluate an AI Agent's Tool Use

An agent can produce the right final sentence after taking the wrong path.

Maybe it called an unauthorized tool and recovered. Maybe it queried five systems when one read would do. Maybe the final answer was correct only because a tool error leaked the expected value. Final-output grading misses all of that.

Agent evaluation needs two views:

• Outcome: Did the user get the correct result?
• Trajectory: Did the agent use an acceptable sequence of decisions and tools?

The second view is what separates agent evaluation from ordinary response evaluation.

Capture a trajectory the evaluator can inspect

Represent each meaningful step as structured data:

{
  "trace_id": "trace_204",
  "steps": [
    {
      "type": "tool_call",
      "tool": "order_get",
      "arguments": {"order_id": "ord_81ks92"},
      "result": {"status": "delayed"},
      "latency_ms": 184
    },
    {
      "type": "tool_call",
      "tool": "replacement_create",
      "arguments": {"order_id": "ord_81ks92"},
      "result": {"status": "approval_required"}
    },
    {
      "type": "approval_request",
      "policy": "replacement-v2"
    }
  ]
}

Do not require the evaluator to parse console logs. Stable step types, tool names, argument schemas, result codes, and parent-child relationships make evaluation cheaper and less ambiguous.

The LLM observability trace schema covers the telemetry foundation. Evaluation turns those traces into judgments.

Grade six independent dimensions

Keep these scores separate. One blended score hides the repair.

A selection failure points toward tool names, descriptions, catalog size, or routing. An argument failure points toward schemas, missing context, or clarification. An order failure points toward orchestration and policy.

Use deterministic checks wherever possible

Code evaluators are reliable for hard constraints:

Python

def no_unapproved_refund(trace):
    approved = set()
    for step in trace.steps:
        if step.type == "approval" and step.decision == "approved":
            approved.add(step.tool_call_id)
        if step.type == "tool_call" and step.tool == "refund_order":
            if step.id not in approved:
                return False
    return True

Other deterministic assertions:

• forbidden tools were never called;
• required tool was called exactly once;
• arguments satisfy the schema;
• resource identifiers match the authenticated user;
• approval precedes the side effect;
• retry count stays under the limit;
• operation keys remain stable across attempts;
• final structured output matches required fields.

Use semantic graders for questions such as whether the chosen plan was sensible or the final explanation was helpful. Do not ask an LLM judge to grade conditions code can establish exactly.

Allow more than one valid path

Exact sequence matching is brittle. These trajectories may both be valid:

customer_get → order_list → order_get

order_search_by_email → order_get

Define required invariants and acceptable partial orders instead:

- order identity must be established before replacement creation
- replacement creation requires approval
- customer-visible confirmation happens only after a receipt exists

For narrow, regulated workflows, an exact approved path may be appropriate. For research and troubleshooting agents, evaluate constraints and outcomes while allowing exploration.

Test abstention and clarification

Tool selection accuracy must include “no call.” Otherwise the agent learns that doing something always scores better than asking or stopping.

Add cases where:

• the answer is already in the conversation;
• the user lacks permission;
• a required identifier is ambiguous;
• the requested capability does not exist;
• the task is outside policy;
• the external dependency is unavailable;
• a consequential action lacks explicit intent.

Expected behaviors may be a direct answer, clarification, refusal, delayed retry, or human intervention. Label the expected control flow, not only the final prose.

Build error-recovery cases on purpose

Inject structured failures:

This tests whether the runtime and model share the same error vocabulary. The error contracts in AI Agent Tool Design make those expectations explicit.

Score efficiency without rewarding shortcuts

Useful efficiency measures:

• tool calls per successful task;
• duplicate calls;
• tokens and cost per successful task;
• wall-clock latency;
• unnecessary retrieval volume;
• retries by error class;
• loop depth;
• percentage of calls that changed workflow state.

Set safety and correctness gates first. Then compare efficiency among runs that pass. A one-call trajectory that skips authorization is not an optimization.

Create a dataset from three sources

Designed cases

Cover permissions, required tools, prohibited tools, ambiguous inputs, and critical business rules before launch.

Production failures

Turn corrected traces into regression cases. Preserve the input, environment fixture, expected invariants, and reviewer explanation.

Controlled synthetic variations

Generate paraphrases, reordered details, missing fields, and tool-result variations. Review samples and keep them in a separate split so synthetic volume does not drown out real behavior.

Version the tool catalog and environment fixture with the dataset. A trajectory cannot be judged fairly against tools that were unavailable when it ran.

A release rubric

Critical gates
- zero unauthorized side effects
- zero cross-tenant access
- all required approvals observed

Quality thresholds
- tool selection ≥ 95%
- argument validity ≥ 98%
- task completion ≥ 90%
- safe recovery ≥ 95%

Efficiency guardrails
- no more than 1 duplicate call per 100 tasks
- P95 calls per successful task may not regress by >10%
- cost per successful task stays within budget

Segment by task type and severity. A healthy overall average can hide a broken refund or access-control path.

Agent trajectory checklist

• Traces expose structured steps, arguments, results, and relationships.
• Selection, arguments, order, recovery, efficiency, and outcome are separate.
• Deterministic rules grade hard constraints.
• Multiple valid paths are represented as invariants where appropriate.
• No-tool, clarification, refusal, and approval cases are included.
• Failures and uncertain writes are injected deliberately.
• Efficiency is compared only after safety and correctness pass.
• Production failures become regression tests.
• Dataset, tool catalog, policies, and environment fixtures are versioned.
• Release gates are segmented by task and severity.

Sources and further reading

• LangSmith application-specific evaluation approaches
• LangSmith intermediate-step evaluation
• Anthropic: Writing effective tools for AI agents