AI Agent Regression Testing in CI

An AI-agent CI gate should block known behavioral regressions without pretending stochastic outputs are ordinary string snapshots. Pin the application configuration, run deterministic invariants on every case, score semantic outcomes on a versioned dataset, compare against the accepted baseline, enforce cost and latency budgets, and preserve failing traces for review.

Test the trajectory as well as the final answer. A correct refund produced without authorization is still a failed release.

Divide the suite by certainty and cost

Keep the pull-request gate small enough to run consistently. A test nobody runs because it takes an hour has no protective value.

Version the complete evaluation envelope

A reproducible run needs more than a prompt:

{
  "app_commit": "4a921d7",
  "dataset": "support-regression-v12",
  "prompt_bundle": "support-31",
  "model_route": "route-8",
  "tool_catalog": "tools-14",
  "policy": "policy-6",
  "retrieval_index": "kb-2026-08-25",
  "evaluator_bundle": "eval-9",
  "fixture_seed": "ci-8421"
}

If a provider does not expose an immutable model version, record the exact model identifier, region, request settings, and run time. Do not call a failed comparison fully reproducible when a critical dependency can move silently.

The agent trajectory evaluation rubric defines selection, arguments, order, recovery, efficiency, and outcome as separate dimensions.

Prefer deterministic invariants

Use code for rules code can decide:

• no write tool before authorization;
• no resource outside the fixture tenant;
• valid arguments for every tool call;
• one business effect per operation key;
• terminal success only with a durable receipt;
• maximum tool calls, tokens, spend, and wall time;
• citations resolve to retrieved evidence;
• secrets and prohibited fields never appear in output.

Use semantic evaluators only for qualities such as usefulness, correctness, groundedness, or tone that cannot be expressed reliably as invariants. Calibrate model judges against reviewed examples and retain the judge rationale, but do not treat the judge as ground truth.

Gate on cases and slices, not one average

A useful result file contains both case-level and aggregate decisions:

The percentages are examples; choose them from product risk and measurement variance. A global 90% score can hide a catastrophic drop in a small high-stakes slice.

For stochastic evaluators, repeat only the cases near the decision boundary or use an explicitly defined sampling plan. Never rerun the full suite until random variation produces a green result.

A CI workflow skeleton

name: agent-quality
on: pull_request

permissions:
  contents: read

concurrency:
  group: agent-eval-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  evaluate:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install --require-hashes -r requirements-eval.txt
      - run: pytest tests/agent_contracts -q
      - run: python evals/run.py --suite pr --output artifacts/results.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: agent-eval-results
          path: artifacts/

Pin third-party actions to full commit SHAs in a hardened implementation. Use an isolated test account, least-privilege credentials, capped provider budgets, and fixtures that cannot reach production. GitHub documents that pull requests from forks normally receive restricted tokens and no secrets; avoid workflows that elevate untrusted code.

Make failure artifacts useful

For each failed case, save a redacted packet:

case ID and slice
configuration envelope
expected invariants and outcome
actual terminal outcome
ordered trajectory with tool statuses
evaluator results and rationale
token, cost, and latency totals
first divergent step

Do not upload raw customer prompts or credentials. Artifact access and retention are security controls, not CI implementation details.

Control flakes instead of normalizing them

Classify a flaky test before changing the threshold:

1. Uncontrolled fixture: time, randomness, external data, or shared state moved.
2. Provider variability: model or service behavior changed.
3. Evaluator instability: judge instructions or evidence are ambiguous.
4. Real product variance: the agent has multiple valid trajectories.

Fix fixtures, pin what can be pinned, and allow multiple valid paths through invariants. Quarantine only with an owner and expiry. A permanently ignored test is deleted protection.

MLflow’s current documentation distinguishes dataset evaluations, which measure aggregate behavior, from regression tests, which gate named failures. That is a useful framework regardless of tooling: a production bug becomes a binary case, while broad quality remains a measured distribution.

Promotion checklist

• Critical behavior has deterministic invariants.
• Every production failure becomes a named case.
• Dataset, app, model route, tools, policy, and evaluators are versioned.
• High-risk slices cannot hide inside an average.
• Cost and latency are measured per successful task.
• CI uses isolated fixtures and least-privilege credentials.
• Failing traces are saved safely and are reviewable.
• Baseline changes require an explicit, documented decision.

Use the AI agent failure taxonomy to label the first divergent contract, then encode the repair here. For MCP-based agents, add the protocol-specific MCP server test harness.

For a smaller dependency-free starting point, download the tested Python LLM evaluation harness, then replace its replay adapter with your application and add trajectory invariants.

Sources and further reading

• MLflow: Regression testing and CI/CD for agents
• MLflow: Building agent and LLM evaluation datasets
• GitHub Actions workflow syntax and permissions
• GitHub Actions workflow artifacts