An LLM judge is a measurement instrument. It is not ground truth.
Use it when a deterministic check cannot capture the behavior and human review cannot cover every run. Before trusting its score, prove that it agrees with reviewed examples often enough for the decision you plan to automate.
Use the cheapest valid evaluator
Choose evaluators in this order:
| Behavior | Preferred evaluator |
|---|---|
| Valid JSON or schema | parser and schema validator |
| Exact ID, amount, or category | deterministic comparison |
| Required citation exists | programmatic check |
| Claim supported by evidence | LLM judge with cited rationale |
| Tone or usefulness | rubric-based LLM or human reviewer |
| High-stakes ambiguous case | human review |
Do not ask a model whether JSON is valid. Do not use string equality to judge a helpful explanation. Match the evaluator to the property.
The evaluator overview in Evaluating LLM Outputs provides the wider testing strategy. This article focuses on model graders.
Write an observable rubric
“Good answer” is not a rubric. Define evidence a reviewer can point to.
Groundedness2 = Every factual claim is supported by supplied evidence.1 = Main conclusion is supported; a non-critical claim is unsupported.0 = A central claim is unsupported or contradicts the evidence.
Give critical errors their own fail rule. A response should not average its way past a privacy leak because the prose was relevant and concise.
Ask the judge to return structured output:
{"score": 1,"critical_failure": false,"claims": [{"text": "...", "evidence_id": "doc-4:12", "verdict": "supported"}],"reason": "One secondary claim lacks evidence."}
The rationale makes disagreement review possible. It should not be shown as hidden model reasoning; request a concise evidence-based justification.
Build a calibration set
Take 50–100 cases spanning scores, categories, and risk levels from the evaluation dataset. Have qualified humans label them without seeing the judge result.
Measure more than raw agreement:
- confusion matrix for pass and fail;
- false-pass rate, especially on critical cases;
- agreement by category and language;
- score distance for ordinal rubrics;
- human-human agreement as context;
- rate of invalid or unparseable judge outputs.
If the judge is used as a release gate, false passes usually cost more than false failures. Set the threshold accordingly.
Test known sources of bias
Run controlled variants:
Position
For pairwise comparisons, swap answer A and B. The winner should not change merely because its position changed.
Verbosity and style
Compare a concise correct answer with a longer answer containing the same facts. A correctness judge should not reward polish or length unless the rubric says so.
Self-preference
Include outputs from the same model family and other families. Report whether the judge disproportionately favors one source.
Reference leakage
Check whether copying phrases from the reference earns a high score despite factual errors elsewhere.
Prompt injection
Place adversarial instructions inside the candidate answer and retrieved evidence. The judge must treat them as data, not instructions.
The attack model overlaps with prompt-injection testing for AI agents.
Keep the judge stable and versioned
Pin the judge prompt, rubric, model/deployment ID, sampling settings, and output schema. Recalibrate when any of them changes.
Run repeated judgments on a sample to quantify instability. Majority voting can reduce variance, but it multiplies cost and does not remove systematic bias.
Store the judge result beside the application trace:
case_id, output_hash, judge_version, score, rationale,human_label, disagreement_status, timestamp
Route uncertainty to people
Human review is most valuable for:
- scores near the release threshold;
- critical-risk cases;
- judge-human disagreements;
- new categories absent from calibration;
- conflicting or insufficient reference evidence;
- suspected evaluator attacks.
Sample clear passes and failures too. Otherwise drift can grow unnoticed outside the review band.
Judge readiness checklist
- Deterministic checks handle deterministic properties.
- Rubric levels describe observable evidence.
- Critical failures cannot be averaged away.
- Calibration labels were created independently by qualified reviewers.
- False-pass rate is measured by category and risk.
- Position, verbosity, self-preference, leakage, and injection are tested.
- Judge prompt, model, schema, and settings are versioned.
- Uncertain and high-risk cases route to human review.
