Human Evaluation for LLMs: Annotation and Agreement

Human evaluation is useful when the quality criterion requires judgment that deterministic tests cannot capture: usefulness, tone, completeness, nuanced policy application, or whether an answer supports the user's actual task. It is not automatically a gold standard. Vague criteria, biased assignments, inconsistent annotators, and hidden adjudication can turn expensive labels into false confidence.

A defensible process defines the decision, samples representative outputs, calibrates annotators, measures disagreement, adjudicates critical cases, and reports limitations.

Start from the decision

Write the release or research decision before writing the rubric:

Decision: release candidate B only if it is non-inferior on correctness,
improves task usefulness on the target slice, and has no critical policy failures.

This prevents an evaluation from collecting many pleasant-looking scores without a clear use. Define which errors are critical, which dimensions may trade off, and what uncertainty would require more samples.

Separate the criteria

Avoid one “overall quality” score. Two annotators can agree that an answer is poor while disagreeing about why. Use criteria with observable anchors.

Use only dimensions that influence the decision. Define each label using positive and negative examples. “Helpful” is not an operational definition.

Build an annotation guide

Every guide should contain:

1. the task and intended user;
2. the evidence or reference an annotator may use;
3. definitions and boundary examples for every label;
4. precedence rules when criteria conflict;
5. how to mark missing context or an invalid item;
6. what constitutes a critical error;
7. when to abstain or escalate;
8. examples that are not part of the measured set.

Keep the source, model identity, and experiment condition hidden when possible. Randomize response order in pairwise comparisons. Record the assignment seed and interface version.

Choose the comparison format

Categorical labels work well for concrete failures such as supported/unsupported or acceptable/critical. They are easy to gate but may lose nuance.

Ordinal scales capture degrees of quality. Anchors must describe observable differences; a bare 1–5 scale invites personal interpretation.

Pairwise comparison asks which of two outputs better satisfies a criterion. It can be easier than absolute scoring, but order and verbosity bias still matter. Include “tie” or “both unacceptable” when appropriate.

Error tagging identifies failure mechanisms and is valuable for repair. Allow multiple tags only when the analysis can use them.

Use a primary decision label plus a small number of diagnostic labels rather than asking annotators to answer twenty questions per item.

Sample the work users actually see

Stratify the dataset by important conditions:

• user intent and task complexity;
• language, region, or user group where relevant;
• answerable and unanswerable cases;
• normal, edge, and adversarial inputs;
• source quality or tool availability;
• long-context or later-conversation turns;
• known production failure categories.

Do not let easy high-volume traffic hide a rare critical slice. Set minimum sample sizes per release-critical slice and report each slice separately.

Calibrate before measuring

Run a pilot on 10–30 items that will not enter the final result. Annotators label independently, then discuss disagreements using the guide. Update ambiguous definitions and examples. Repeat until the remaining disagreements reflect genuine judgment rather than missing instructions.

Calibration is training, not a chance to force unanimous opinion. Preserve valid ambiguity and allow abstention.

Measure agreement correctly

Report raw agreement and a chance-corrected or reliability statistic suited to the label type and assignment design. Cohen's kappa is limited to two raters; Fleiss' kappa supports multiple raters for categorical labels; Krippendorff's alpha supports several measurement levels and missing assignments. Weighted statistics may suit ordinal labels.

Do not use a universal threshold such as “0.8 means good” without context. Agreement depends on prevalence, category balance, task ambiguity, and statistic choice. A high raw agreement with a low kappa can occur when one label dominates.

Report:

items and slice counts
annotators per item and annotator qualifications
label distribution
raw agreement
agreement/reliability statistic with uncertainty when feasible
abstention and invalid-item rate
adjudication rate
rubric and interface version

The ACL-published best-practices review for generated-text evaluation recommends defining separate criteria, using multiple annotators for qualitative analysis, reporting agreement, and controlling order and learning effects.

Adjudicate without hiding disagreement

Use adjudication for critical errors, reference defects, and labels that feed a release gate. The adjudicator should see the original independent labels and reasons, apply the same guide, and record the resolution and cause.

Track disagreement categories:

• unclear criterion;
• missing or conflicting reference;
• annotator mistake;
• genuinely subjective tradeoff;
• interface or assignment error.

Update the guide for systematic ambiguity, then relabel affected items if the change alters meaning. Do not overwrite the original labels; store adjudication as another layer.

Annotation record schema

{
  "item_id": "support-0042",
  "dataset_version": "support-eval-7",
  "rubric_version": "answer-rubric-3",
  "condition_blind_id": "candidate-b",
  "annotator_id": "reviewer-12",
  "assignment_seed": 7319,
  "labels": {
    "correctness": "minor",
    "groundedness": "fully",
    "policy": "pass",
    "usefulness": 2
  },
  "reason": "Correct process; omitted the stated response deadline.",
  "abstained": false,
  "created_at": "2026-08-25T09:00:00Z"
}

Store references and model outputs separately with access controls. Do not expose sensitive production conversations to annotators without a lawful, minimized process.

Use human labels to calibrate automation

Human evaluation is expensive, so teams often train or prompt an automated judge. Compare the judge against held-out human labels by criterion and slice. Inspect false passes on critical errors, not only overall agreement. Recalibrate after judge-model, rubric, or task changes.

Microsoft's rubric guidance treats human judgments as the reference for refining automated judges. That does not make every human label correct; the protocol, agreement, and adjudication remain part of the evidence. See the LLM-as-judge calibration guide for the automated layer.

Release checklist

• The decision and critical failures are defined before labeling.
• Criteria are separate, observable, and illustrated.
• Outputs are blinded and randomized where possible.
• Dataset slices represent users and known failures.
• Annotators complete a calibration round.
• At least two independent labels cover critical items.
• Agreement, abstention, and adjudication are reported.
• Original labels remain available after adjudication.
• Sensitive data is minimized and access-controlled.
• Automated judges are validated on held-out human labels.

Start with the LLM evaluation dataset guide, then add this annotation protocol only where deterministic scoring is insufficient. For conversations, extend the unit of analysis with the multi-turn LLM evaluation schema.

Sources and further reading

• ACL Anthology: best practices for human evaluation of generated text
• NIST ARIA pilot evaluation report
• Microsoft: rubric and human-judgment reference
• NIST AI Risk Management Framework