Offline vs Online LLM Evaluation

Offline evaluation answers “should this version ship?” on a controlled, versioned dataset. Online evaluation answers “how is the deployed system behaving for real traffic?” on sampled traces, operational signals, and user outcomes. A production system needs both, connected by a promotion loop that turns reviewed online failures into offline regression cases.

Online evaluation does not replace a release gate, and offline scores do not prove production health.

Use the modes for different unknowns

Run deterministic policy, schema, authorization, and state checks in both modes when possible. Semantic qualities need calibrated evaluators and periodic human review.

The closed feedback loop

curated dataset → candidate offline eval → release decision → canary
       ↑                                                   ↓
regression case ← reviewed failure ← sampled online traces + feedback
       ↑                                                   ↓
evaluator calibration ← expert labels ← drift and disagreement analysis

Each arrow needs an owner and a recorded decision. A dashboard that never changes the dataset or release policy is observation, not a learning system.

Design the offline gate

Build the dataset from:

• critical product requirements and prohibited behavior;
• reviewed representative tasks by traffic segment;
• difficult edge cases and expected abstentions;
• every corrected production incident;
• controlled adversarial and dependency-failure fixtures.

Version the dataset, prompt, model route, retrieval index, tools, policy, and evaluator bundle. Compare the candidate with the accepted baseline on the same envelope.

Use hard gates for deterministic critical behavior and slice-level thresholds for semantic quality. The LLM evaluation dataset guide covers case construction; the agent regression CI workflow shows how to operationalize hard gates.

Design online evaluation as sampling

You usually cannot send every production trace to expensive judges or humans. Define a sampling plan instead:

Keep selection metadata so analysts do not mistake a failure-enriched sample for the population rate. When estimating a population metric from unequal sampling probabilities, use appropriate weighting and report uncertainty.

Separate synchronous controls from asynchronous scoring

Online evaluation has two paths:

Synchronous controls block or transform a response before effect: schema validation, authorization, policy rules, tool limits, citation ID integrity, and deterministic redaction.

Asynchronous evaluation scores traces after the response: semantic correctness, groundedness, relevance, conversation completeness, user frustration, and emerging failure analysis.

Do not put a slow model judge in the critical path unless the risk case specifically requires it and the latency budget includes it. An asynchronous judge can alert and create cases, but it cannot undo a completed side effect.

MLflow’s current automatic-evaluation documentation makes the same operational distinction: production judges run asynchronously over traces, while offline evaluation runs manually against curated data for regression and version comparison.

Keep evaluator behavior under evaluation

Use the same rubric definitions across offline and online modes where the tasks match. Then measure evaluator agreement against reviewed examples:

agreement by label and slice
false-pass rate on critical failures
false-fail rate on acceptable variation
abstention/ungradable rate
cost and latency per assessment

Recalibrate when the application domain, evidence format, or traffic language changes. Do not tune a judge only against cases from one model version and assume it transfers.

Promotion rules from online to offline

Promote a production trace when it is:

• a confirmed harmful or costly failure;
• a new high-volume intent;
• an evaluator disagreement resolved by a reviewer;
• a valid behavior the current dataset wrongly rejects;
• a rare boundary with large consequences;
• evidence of a shifted source, tool, or policy contract.

Before storing it, redact or synthesize sensitive fields under policy, preserve the relevant fixture and version envelope, add an expected outcome/invariants, and name the reviewer. Raw logs are not automatically good test cases.

Operating scorecard

Checklist

• Offline and online modes have different documented decisions.
• The offline envelope is versioned and compared with a baseline.
• Online sampling probabilities and filters are retained.
• Critical controls run synchronously at enforcement points.
• Semantic production scoring is asynchronous and cost-bounded.
• Evaluators are calibrated against named human judgments.
• Confirmed online failures become offline regression cases.
• Privacy, retention, and access policies cover trace-derived datasets.

Use the production observability schema to collect evaluable traces and the broader LLM output evaluation guide to choose deterministic, model, and human evaluators.

When quality depends on accumulated context or recovery over several turns, use the multi-turn conversation evaluation guide instead of sampling isolated responses.

Sources and further reading

• MLflow: Automatic versus offline evaluation
• MLflow: LLM and agent evaluation lifecycle
• MLflow: Building evaluation datasets from traces and reviewed cases