Evaluating Multi-Turn LLM Conversations

A conversation can contain individually acceptable responses and still fail as a whole. The system may forget a constraint, repeat a question, contradict an earlier answer, expose the wrong state, recover poorly from a tool error, or finish without completing the user's task. Multi-turn evaluation therefore needs three views: each turn, the trajectory, and the final session outcome.

Do not concatenate a transcript, ask for one quality score, and call the result complete.

Define the evaluation unit

Use the view that matches the decision:

Microsoft's current agent evaluation guidance similarly distinguishes last-turn, full-conversation, and per-turn splits. Use all three when a release could improve the final answer while worsening the path.

Write a conversation contract

For each scenario, define:

initial user goal
facts and permissions available to the system
facts revealed at specific turns
required constraints to remember
allowed and forbidden actions
expected state transitions
acceptable clarification questions
completion and escalation conditions
critical failure conditions

The contract should allow more than one good wording or path when the task does. Avoid overfitting to one reference transcript.

Dataset schema

{
  "conversation_id": "return-017",
  "dataset_version": "support-mt-4",
  "scenario": "damaged item with policy exception",
  "slice": ["state-update", "tool-error", "approval"],
  "initial_state": {"order_status": "delivered", "region": "EU"},
  "turns": [
    {
      "turn_id": 1,
      "role": "user",
      "content": "The package arrived damaged.",
      "facts_revealed": ["damage_reported"]
    }
  ],
  "required_facts": ["order_id", "damage_evidence"],
  "forbidden_actions": ["refund_without_authorization"],
  "success": {"state": "approved_resolution_recorded"},
  "critical_failures": ["cross_customer_disclosure", "duplicate_refund"],
  "rubric_version": "support-session-2"
}

Store generated traces separately with model, prompt, tool, policy, and state-store versions. Preserve timestamps and tool attempt identifiers so the trajectory can be replayed.

Build the failure slices

Single-turn examples rarely expose conversational state errors. Include:

• recollection: a fact from an earlier turn becomes necessary later;
• revision: the user corrects or changes a constraint;
• refinement: the user narrows or expands the request;
• reference resolution: “that one” or “the earlier option” must be resolved;
• contradiction: new information conflicts with prior state;
• topic return: the user returns to a suspended subtask;
• tool failure: a transient or permanent error changes the path;
• handoff: another agent or human receives the right context;
• unanswerable: the session should end in refusal or escalation;
• long horizon: the relevant fact is separated by many distractor turns;
• adversarial state: content tries to alter authority or another user's state.

MT-Eval categorizes conversation patterns including recollection, expansion, refinement, and follow-up; MultiChallenge emphasizes realistic instruction and context failures. Use public benchmarks for mechanisms, but create application-specific sessions for release decisions.

Score turns and sessions separately

Turn-level metrics

• factual or task correctness;
• groundedness and citation correctness;
• tool selection and argument validity;
• state read/write correctness;
• policy compliance;
• response relevance and clarity;
• latency and cost.

Trajectory metrics

• invalid or unauthorized action count;
• repeated/no-progress step count;
• unnecessary clarification count;
• first failed transition;
• recovery success after injected failure;
• total tool calls, model calls, time, and cost;
• whether approval occurred before the exact side effect.

Session metrics

• task completion or correct escalation;
• all required subgoals completed;
• final state correctness;
• unresolved contradiction count;
• user effort, measured by avoidable turns;
• critical failure rate.

Do not average critical security failures into conversational fluency.

Use strict joint success where appropriate

For workflows, session success may require all critical conditions:

joint_success =
  task_completed
  AND final_state_correct
  AND no_forbidden_action
  AND no_cross-user disclosure
  AND required_approval_present

Joint success will be lower than the average of component scores. That is useful: one broken authority boundary can invalidate an otherwise smooth conversation.

Generate conversations without hiding assumptions

Use three sources:

1. reviewed production conversations, minimized and de-identified;
2. expert-authored scenarios based on task and incident taxonomies;
3. simulated users for coverage and controlled perturbations.

For simulation, specify the user goal, private state, behavior, information-release schedule, stopping rule, and prohibited cooperation. Validate a sample manually. A simulated user that eagerly reveals everything creates an unrealistic easy test.

Run the same scenario with several generation seeds when path variability matters. Store the seed and simulator version.

Evaluate automated judges at the right scope

A per-turn judge can miss delayed contradictions and task abandonment. A session judge can identify global problems but may fail to localize them or overlook one critical tool call in a long transcript.

Combine:

• deterministic checks for tool calls, state, approvals, and forbidden actions;
• turn judges for answer properties;
• trajectory checks for sequence and budgets;
• session judges or humans for coherence, completeness, and task outcome.

Calibrate model judges against independently reviewed conversations by failure slice. Use the human evaluation protocol for labels that become the reference set.

Diagnose the first broken transition

When a session fails, find the earliest turn where actual state diverged from an acceptable state:

1. Was prior context stored and retrieved correctly?
2. Did the system interpret the new message correctly?
3. Did routing select the right next capability?
4. Were tool arguments and authorization valid?
5. Did tool results update durable state?
6. Did the response reflect the resulting state?
7. Did later turns compound or recover from the error?

Fixing the final response without repairing the first broken transition creates a brittle exception.

Example release gate

- Zero critical authorization or cross-user failures.
- Joint session success must not regress globally or on any critical slice.
- State-revision and tool-error recovery must meet their agreed thresholds.
- P95 session latency and cost per successful session remain within budget.
- Changed failures are reviewed, not only aggregate scores.
- Every production incident becomes a versioned conversation regression.

Compare paired sessions when possible and report uncertainty. A session dataset with 100 conversations does not provide 800 independent samples merely because it contains 800 turns.

Implementation checklist

• Conversation, turn, and state identifiers are stable.
• Scenario contracts define facts, permissions, and completion.
• Failure slices include revision, long-horizon, errors, and unanswerable tasks.
• Turn, trajectory, and session results remain separate.
• Deterministic checks cover authority and side effects.
• Simulated users have controlled goals and information schedules.
• Judge calibration includes complete conversations and critical slices.
• The first broken transition is recorded for each failed session.
• Release gates use joint success and critical-failure rules.
• Production sessions are minimized and access-controlled.

Begin with the LLM evaluation dataset guide, then use offline versus online evaluation to connect conversation regressions to monitored product outcomes.

Sources and further reading

• Microsoft Agent Framework: conversation evaluation strategies
• Microsoft: evaluating conversations at session level
• MT-Eval: multi-turn capabilities benchmark
• MultiChallenge: realistic multi-turn conversation benchmark
• MTRAG: multi-turn conversational RAG benchmark