LLM Evaluation Harness in Python

An LLM evaluation harness is ordinary test infrastructure around a probabilistic component. It loads versioned cases, calls the system under evaluation, runs explicit graders, retains case-level evidence, summarizes meaningful slices, and exits nonzero when a release rule fails.

Start small. The downloadable JoinAI reference uses only Python's standard library and includes working tests. It deliberately does not hide application decisions behind a framework.

Download the working reference

• harness.py
• sample.jsonl
• test_harness.py
• README

Run it with Python 3:

python3 harness.py sample.jsonl --report report.json --min-pass-rate 1.0
python3 -m unittest -v test_harness.py

The sample ships with three passing cases. The test suite covers combined checks, candidate exceptions, summaries, and invalid dataset records.

Architecture

versioned JSONL cases
  -> loader and schema checks
  -> candidate adapter
  -> deterministic/application graders
  -> case-level results
  -> slice summaries
  -> release policy and process exit code

Keep the layers separate. Changing a model adapter should not silently change the dataset or grader. Changing a rubric should create a new evaluator version.

Define the case contract

The reference accepts one JSON object per line:

{
  "id": "safe-refusal",
  "input": {"question": "What is my current account balance?"},
  "candidate_output": "I cannot access a live account balance.",
  "expected": {
    "contains": ["cannot access", "live account balance"],
    "forbidden": ["€", "$"]
  },
  "slice": ["unanswerable", "live-data"]
}

In a real harness, candidate_output is replaced by a call to the application. Keep expected behavior and representative input in the case; keep secrets, live credentials, and mutable environment state elsewhere.

Add a version envelope to every run:

{
  "dataset": "support-eval-7",
  "candidate": "support-service-git-sha",
  "model": "recorded-provider-deployment",
  "prompt": "support-answer-v12",
  "tools": "support-tools-v4",
  "corpus": "help-center-2026-08-25",
  "grader": "deterministic-v2"
}

Without those versions, a score cannot be reproduced.

Use deterministic graders first

The reference implements exact match, required substrings, forbidden substrings, and regular-expression checks. These are appropriate for IDs, schemas, policy phrases, forbidden disclosure, refusal markers, or other concrete contracts.

Python

def grade(case, output):
    expected = case.get("expected", {})
    checks = {}
    if "exact" in expected:
        checks["exact"] = output.strip() == expected["exact"].strip()
    checks["contains_all"] = contains_all(output, expected.get("contains", []))
    checks["excludes_all"] = excludes_all(output, expected.get("forbidden", []))
    checks["pattern"] = matches_pattern(output, expected.get("pattern"))
    return all(checks.values()), sum(checks.values()) / len(checks), checks

Do not use exact string match for a task with many valid phrasings. Add structured parsers, domain rules, retrieval metrics, human labels, or calibrated model judges according to the task. OpenAI's grader API similarly distinguishes string checks, text similarity, and model-based scoring; the right grader depends on the criterion.

Adapt the candidate safely

Replace replay_candidate with an application adapter:

Python

def candidate(case):
    response = support_service.answer(
        question=case["input"]["question"],
        tenant_id=TEST_TENANT,
        request_id=f"eval:{case['id']}"
    )
    return response.text

Use a dedicated test environment. Bound timeouts and retries, derive authorization from the test fixture rather than model output, and make side effects disabled or idempotent. Store tool traces separately if trajectory correctness matters.

The reference catches candidate exceptions and records them as failed results. One broken case should not erase the evidence from every other case.

Report cases before aggregates

The JSON report includes every check, score, latency, and exception. Preserve it as a CI artifact. Aggregate only after case results exist.

Add slice summaries for categories such as:

• answerable versus unanswerable;
• permissions and tenant isolation;
• retrieval, tool use, or structured output;
• language and region;
• routine, edge, adversarial, and incident regressions;
• model/provider route.

A 95% global pass rate can hide a 0% authorization slice. Critical slices need independent gates.

Define release policy in code

The reference supports a global --min-pass-rate, but production policy should express invariants:

Python

policy = {
    "global_min": 0.95,
    "critical_slices": {
        "authorization": 1.0,
        "cross_tenant": 1.0,
        "unsafe_action": 1.0
    },
    "max_p95_latency_ms": 2500,
    "max_cost_per_success": 0.03
}

Compare the candidate with the current production baseline. A fixed threshold alone may approve a regression that remains above the floor.

Add CI

A GitHub Actions step can run the tests and harness:

- name: Test evaluation harness
  run: python3 -m unittest -v test_harness.py
- name: Run offline evaluation
  run: python3 harness.py cases.jsonl --report report.json --min-pass-rate 0.95
- name: Preserve report
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: llm-evaluation-report
    path: report.json

Pin dependencies when you add them. Do not let a CI evaluation call an unversioned production corpus or unrestricted side-effect tool.

Grow the harness only from measured needs

Add features in this order:

1. dataset and version validation;
2. application-specific deterministic graders;
3. slice summaries and baseline comparison;
4. trace capture for retrieval/tools;
5. concurrency with explicit rate and cost budgets;
6. human review workflow;
7. calibrated model judges;
8. online failure ingestion and regression promotion.

EleutherAI's lm-evaluation-harness is a mature open-source project for evaluating language models across many tasks. Use it when its model/task abstraction fits. The smaller JoinAI harness targets application behavior and is intentionally easy to inspect and replace.

Readiness checklist

• Every case has a stable ID, slice, and expected behavior.
• Dataset, system, model, prompt, tool, corpus, and grader versions are recorded.
• Candidate errors become results rather than aborting the run.
• Critical boundaries use deterministic checks where possible.
• Case results are retained before aggregation.
• Slices have independent thresholds.
• The current baseline and candidate are compared on identical cases.
• CI preserves the report even when the gate fails.
• Sensitive cases are minimized and access-controlled.
• Production failures reproduce on the old version before becoming regressions.

Use the LLM evaluation dataset guide to design cases and human evaluation guidance when labels require judgment. For agent trajectories, extend the adapter using the AI-agent regression testing workflow.

Sources and further reading

• Python unittest documentation
• Python json documentation
• OpenAI grader API reference
• EleutherAI lm-evaluation-harness
• NIST AI RMF: Measure