How to Build an LLM Evaluation Dataset

Start an LLM evaluation dataset with 30 reviewed cases, not 3,000 synthetic prompts.

The first dataset exists to make product expectations executable. It should expose important behavior, common traffic, and costly failures. Size comes later, from real usage and disagreement analysis.

Write the behavior contract first

List what the system must do and must never do. For a support agent:

Must:
- answer account-policy questions from current evidence
- ask for clarification when account identity is ambiguous
- escalate refund exceptions

Must not:
- invent account state
- reveal another customer's information
- execute a refund without approval

Each rule needs at least one normal case and one boundary case. This produces a dataset tied to risk rather than a collection of convenient questions.

The broader guide to evaluating LLM outputs explains grader types. Dataset design comes first because a precise grader cannot repair irrelevant cases.

Use four sources of cases

Use synthetic cases to vary an already understood behavior: typos, languages, reordered facts, longer context, or conflicting instructions. Have a person review them before they influence a release score.

Define a case schema

A row needs more than input and ideal answer:

{
  "case_id": "refund-approval-017",
  "input": {"messages": ["..."]},
  "expected_behavior": "request_approval",
  "reference_facts": ["refund exceeds agent limit"],
  "forbidden_behavior": ["execute_refund"],
  "category": "refunds",
  "risk": "high",
  "difficulty": "policy_exception",
  "source": "incident-284",
  "reviewed_by": "Name",
  "reviewed_at": "2026-08-21",
  "dataset_version": "support-v3"
}

Reference answers are optional. For open-ended tasks, required facts, forbidden claims, and a rubric are often more stable than one “golden” paragraph.

Split by purpose, not randomly

Maintain three sets:

• development: visible to people changing prompts and workflows;
• release regression: stable cases that gate deployment;
• holdout: protected cases used to detect overfitting.

Random row splitting can leak near-duplicates across sets. Group variants from the same source incident or template before splitting.

Do not report one average alone. Segment by category, risk, language, tool path, and difficulty. Ten easy FAQ cases should not drown out one failed authorization case.

Label with disagreement in mind

Create a short annotation guide with examples for each label. Ask reviewers to mark uncertain rather than forcing a verdict.

For subjective dimensions, double-label a sample and inspect disagreement. Disagreement may mean the rubric is vague, the evidence conflicts, or the product has not made a decision. Those are useful findings.

If using an LLM judge, calibrate it against these reviewed labels using the process in LLM-as-Judge: Rubrics, Bias, and Calibration.

Turn production failures into regression cases

The feedback loop should be explicit:

production trace
→ failure triage
→ redact and minimize
→ reviewer labels expected behavior
→ add to candidate dataset
→ verify case catches old behavior
→ promote to release regression set

A regression case that the old system already passes proves nothing. Replay it against the failing version before accepting it.

For retrieval systems, store relevant document and passage IDs as described in RAG Evaluation. This lets one case diagnose retrieval separately from generation.

Version everything needed to reproduce a run

Record dataset version, case hashes, evaluator version, model or deployment ID, prompt version, tool schemas, and corpus/index version. Never silently edit a reference label after a run; create a new dataset version and document why.

Treat eval data as production data. It may contain customer content, personal data, or confidential failures. Apply retention, access controls, and redaction before uploading it to any evaluation service.

Dataset readiness checklist

• Cases map to explicit required and forbidden behaviors.
• Common traffic and high-risk boundaries are represented.
• Synthetic cases are reviewed and labeled by origin.
• Near-duplicates cannot leak into the holdout set.
• Subjective labels include a rubric and uncertainty option.
• Metrics are segmented by risk and category.
• Every incident case reproduces the previous failure.
• Dataset access, retention, and versioning are documented.

Sources

• OpenAI Evals API reference
• OpenAI Files API: eval dataset files and expiration
• NIST AI RMF Playbook: Measure