The harness is a product, not a notebook

An experiment notebook can answer “did this prompt appear better on ten examples?” A harness must answer a stronger question: “may artifact pipeline-2026.08.17 serve traffic for tenant class X, under an explicit latency and cost budget, and can we explain a rollback?”

Treat the following as versioned first-class artifacts:

• the evaluation case and its labels;
• corpus snapshot, document parser, chunking policy, index, and access-control policy;
• retrieval configuration, including query rewriting, filters, top-K, fusion, and reranker;
• context assembly and system prompt;
• model provider, exact model alias or revision, decoding parameters, tool schema, and gateway route;
• evaluator version, rubric, thresholds, and release policy;
• the resulting trace, costs, scores, and human overrides.

Without these identities, a score is a story rather than evidence. A team may attribute a regression to the new model when a nightly reindex changed chunk boundaries, or celebrate an improvement that came from removing difficult cases. Reproducibility is therefore the first feature, not a compliance afterthought.

A useful reference implementation has four planes. The case plane stores immutable test cases, annotations, sensitivity labels, and sampling strata. The execution plane invokes candidate and baseline pipelines with a captured configuration. The judgment plane calculates deterministic metrics, invokes calibrated model judges where appropriate, and manages human adjudication. The decision plane applies gates and publishes a machine-readable verdict to CI, deployment automation, and observability.

Keep raw production requests out of the default golden set. Evaluation data has its own access policy, retention period, and redaction flow. The harness should support a secure enclave for approved, de-identified samples, not quietly duplicate every customer conversation into a developer bucket.

Define the quality contract before the metrics

“Helpful” is not a release criterion. Start by defining user-visible tasks and the damage caused by failure. A support assistant answering a product FAQ can tolerate a polite abstention. A maintenance assistant proposing a lockout procedure cannot invent a step. An internal search tool may value recall; a customer-facing answer service may value citation correctness and safe refusal.

Write each task family as a contract:

1. Input boundary: supported language, tenant, role, document freshness, conversation length, and tool availability.
2. Expected behavior: answer, cite, classify, call a tool, ask a clarifying question, or abstain.
3. Evidence boundary: what sources or structured facts may support the result.
4. Safety boundary: prohibited claims, data categories, actions requiring confirmation, and escalation path.
5. Service boundary: p95 latency, maximum cost, availability, and acceptable fallback behavior.

This makes metrics subordinate to product intent. A factuality score of 0.94 can conceal one unacceptable unsafe instruction. Conversely, a lower answer-completeness score may be a good change if a system learned to refuse when sources are absent. Use separate gates for “must never regress,” “must improve eventually,” and “observe but do not block.”

The harness also needs a baseline. Pin a deployed artifact or a deliberately approved reference configuration. Comparing only a candidate with an absolute threshold is brittle: a new domain or a difficult fresh slice can lower both systems. A paired comparison on the same cases exposes whether the candidate is worse than the service it intends to replace.

Build golden sets as a living, stratified asset

A golden set is not a random export of questions. It is a small, governed representation of decisions the system must get right. Each case needs more than question and expected_answer. At minimum, store a stable case ID, task family, input, required or allowed evidence IDs, expected outcome, evaluation rubric, risk tier, language, metadata filters, and annotation provenance.

For RAG, label evidence at the appropriate granularity. A document-level label is too weak when the answer depends on one row in a table. A span-level citation or canonical fact ID lets the harness distinguish “retrieved the right policy” from “retrieved a related policy.” Where exact wording is not practical, define a set of required claims and disallowed claims.

Stratify intentionally. A credible starter suite includes:

• common successful questions, because most traffic is not adversarial;
• exact identifiers, product codes, dates, negations, and rare acronyms;
• ambiguous queries that should request clarification;
• no-answer cases that must abstain rather than improvise;
• stale-versus-current policy conflicts;
• multilingual and code-switched requests;
• ACL and tenant boundaries, evaluated in an isolated secure environment;
• long-context, table, OCR, and attachment cases;
• high-risk domains where a human review is mandatory;
• production incidents that have been redacted, annotated, and promoted into regression cases.

Do not make the set static. A static benchmark becomes a target for overfitting and gradually stops describing production. Add cases after material incidents, changes in corpus composition, new customer workflows, and human-review disagreements. Retire cases only with an audit trail; deletion can make a chart improve without making a product better.

Split the asset into three populations. The commit suite is fast, deterministic, and inexpensive: perhaps 30–100 critical cases. The release suite is broader and may use a stronger judge or more provider routes. The continuous suite samples held-out and fresh cases after deployment. Never expose every hidden case to developers tuning prompts; preserve a holdout set for decision confidence.

Measure retrieval, context, and generation separately

End-to-end answer scoring is necessary but insufficient. An answer can be wrong because the evidence did not exist, retrieval missed it, reranking buried it, context truncation removed it, or the model ignored it. If the harness reports only “quality fell,” engineers will optimize the most visible component rather than the broken one.

For retrieval, calculate Recall@K: whether at least one required evidence item appeared among the first K candidates. Use MRR when the rank of the first correct result matters, and nDCG when labels have graded relevance. Report these per stratum, not only as an aggregate. A high overall Recall@20 can hide failure on exact IDs or non-English questions.

For context assembly, record evidence coverage after filtering, deduplication, and token truncation. Measure context precision: how much of the provided material was useful, and context recall: whether all required support survived. Inspect citation-span coverage for high-risk claims. This stage often explains why retrieval looks healthy while answer quality falls after a smaller context window or an aggressive compression change.

For generation, use task-specific tests:

• claim support or groundedness: each material claim must be entailed by supplied evidence;
• answer completeness: required claims are present when evidence permits;
• citation correctness: cited spans actually support the corresponding claim;
• instruction and policy compliance;
• tool-call schema validity and task completion for agent workflows;
• calibrated abstention: refusal when evidence is missing, conflicting, unauthorized, or below a confidence threshold.

Exact-match remains valuable for extraction, routing, structured JSON, and tool arguments. It is a bad primary metric for explanatory answers where many phrasings are correct. Model-based judges can evaluate semantic rubrics, but they are measurements with error bars, not ground truth. Calibrate them against human labels, pin the judge prompt and model, use blinded paired comparisons, and periodically recheck disagreement by risk tier.

Design the evaluation pipeline as deterministic infrastructure

The unit of work should be an evaluation run, not an ad hoc script. A run manifest records inputs and outputs so it can be replayed.

run
  -> resolve case-set revision and corpus/index snapshot
  -> invoke baseline and candidate under identical principal and budget
  -> persist stage traces, outputs, token use, provider route, and errors
  -> compute deterministic retrieval and format metrics
  -> run semantic judges only where needed
  -> aggregate by risk tier and traffic stratum
  -> apply release policy
  -> publish verdict, diff, and artifacts

Use content-addressed IDs where possible. A pipeline manifest can reference chunker@sha, embedding@version, index@build-id, prompt@sha, and gateway-policy@revision. Seed any stochastic sampling; record temperature and retries. If an external provider cannot promise deterministic output, rerun a small sentinel set and report variance rather than pretending a one-shot score is stable.

Store one trace row per case and stage. It should reveal the query after rewriting, filters applied, candidate IDs and scores, selected context IDs, final output, citations, tool calls, errors, latency breakdown, and cost breakdown. Protect it as sensitive operational data: hash or redact prompts where needed, retain references to protected evidence instead of copying it, and prevent broad analytics access to private text.

The harness interface should be provider-neutral. Application code calls an adapter that returns a typed trace, not a text string. That design lets the same case execute against two LLM providers, a fallback model, or a local model without changing evaluators. It also aligns with the routing, accounting, and policy layer described in LLM Gateway FinOps in 2026.

Use CI gates that match risk and change size

A gate is a policy decision expressed as code. Avoid a single universal score such as “quality must be above 0.90.” It invites metric gaming and makes harmless variance block releases while serious safety regressions pass on an average.

A practical policy has four outcomes:

• block: a critical case fails, a safety/ACL invariant fails, structured output becomes invalid, or a hard SLO budget is exceeded;
• hold for review: a statistically meaningful regression occurs in a high-risk slice, a judge and human label disagree, or cost rises beyond a review budget;
• warn: a non-critical metric moves within a tolerance band or sample size is too small;
• pass: hard invariants hold and paired quality, latency, and cost criteria meet the defined envelope.

Run a narrow commit gate for prompt, configuration, and code changes. Run the full release suite when changing the corpus parser, chunker, embedding model, retriever, reranker, model route, tool behavior, or authorization logic. An index rebuild is a release candidate even if application code did not change.

Use paired deltas, not only aggregate scores. For every baseline-candidate case pair, classify improved, unchanged, regressed, and incomparable. Require zero regression for a set of critical cases, an allowed regression budget for lower-risk cases, and a minimum net gain where an optimization is claimed. Bootstrap confidence intervals or repeated runs are useful when outputs are stochastic. They do not replace engineering judgment, but they stop teams from declaring victory on noise.

Keep gate configuration in the repository next to service code. A reviewer should see that a change relaxes a threshold, adds a fallback, or excludes a test case. Require an explicit approval for changing critical labels or release policy. This is analogous to changing an authorization rule: legitimate sometimes, never invisible.

Control latency and cost as quality dimensions

An answer that is grounded but arrives after a user has left is not production quality. Neither is a small accuracy gain that doubles cost at the traffic volume of a help center. Every evaluation trace should include end-to-end and stage-level latency, input/output tokens, cache status, reranker calls, tool calls, and normalized currency cost.

Evaluate a quality frontier, not a single winner. For example, compare a fast route without a reranker, a standard route with hybrid retrieval and a compact model, and a premium route with reranking plus a stronger model. Plot or report quality, p50/p95 latency, and cost per completed task for each traffic stratum. The correct policy might route simple FAQ questions to the standard path and reserve the premium route for complex, high-value cases.

Set budgets at more than one level:

• a per-request ceiling to stop runaway tool loops or prompt expansion;
• a p95 latency ceiling by endpoint;
• a cost-per-successful-task target, not merely cost per request;
• an evaluation-run budget so broad tests do not become unaffordable;
• a daily and monthly gateway budget with alerting and safe degradation.

Gateway data is essential here. The gateway should attach route, provider, model, retry count, cache hit, tenant, and policy revision to each trace. The harness then detects a change that “improves” answer quality only because it silently routed every request to the expensive model. It can also test fallbacks deliberately: what happens when a provider times out, a context limit is reached, or the premium budget is exhausted?

Prove behavior in production with shadow traffic

Offline suites are a release filter, not proof that production distribution has not shifted. Shadow traffic safely supplies realism when designed with strict boundaries. Mirror an eligible request to a candidate pipeline, discard its response, and compare traces asynchronously. Do not shadow requests that contain prohibited data unless the candidate runs inside the same approved boundary and policy allows it. Do not repeat side-effecting tool calls; replace them with read-only simulators or recorded fixtures.

Shadow comparisons should answer specific questions. Does the candidate retrieve different documents? Does it abstain more often? Is latency higher at the tail? Does it use more expensive routes? Are citation patterns changing? Sample by tenant, language, task, and risk tier, and cap volume to protect budgets.

After shadowing, use a canary with an explicit rollback trigger. Route a small, eligible cohort to the candidate; measure system health, task outcomes, complaints, human escalations, and safety signals. A feature flag must identify the complete pipeline artifact, not only the model name. On rollback, restore the previous manifest, preserve traces, and promote confirmed incidents to the golden set.

Online feedback is noisy. A thumbs-up favors short agreeable answers; silence does not prove success. Combine it with outcome signals such as successful search refinement, repeated question rate, resolution completion, citation opens, tool success, escalation, and audited human review. The AI observability guide explains the operational telemetry around these signals; the harness turns them into a controlled release loop.

Extend the harness to agents and RAG changes

Agentic systems add state, branching, tool risk, and non-deterministic duration. Evaluate the trajectory as well as the final text. A case can specify initial state, allowed tools, tool fixtures, expected terminal state, maximum steps, forbidden actions, and a budget. Metrics then include task completion, valid tool arguments, policy compliance, loop rate, unnecessary calls, time to completion, and cost.

For RAG changes, make corpus and ACL tests mandatory. A candidate that answers more questions by retrieving documents from another tenant is not an improvement. Include negative authorization cases in a secure suite and assert that unauthorized evidence IDs never reach retrieval candidates, rerankers, context, logs, or model prompts.

Test change combinations selectively. A new embedding model with an old index is an invalid configuration; a new chunker plus new reranker may hide attribution. Run isolated comparisons first, then a combined candidate once individual changes meet their contracts. This discipline is especially important for autonomous workflows discussed in Agentic Engineering in 2026: a small retrieval regression can become a large action error after several tool steps.

Operate the harness as an engineering service

Assign owners. Product and domain experts own task definitions and risk classification. Platform engineers own execution, storage, policy enforcement, and cost controls. Application teams own candidate manifests and regression fixes. A review group owns critical-case changes and human adjudication. “Everyone owns quality” usually means no one can explain a failing gate.

Start smaller than the architecture diagram. In the first month, instrument one RAG endpoint, choose twenty critical cases and fifty representative cases, capture stage traces, and make format, ACL, and no-answer tests blocking. In the second month, add retrieval metrics, paired baseline comparison, a release suite, and gateway cost accounting. In the third, add shadow traffic, canary controls, human review sampling, and an incident-to-golden-set workflow.

Review the suite monthly. Look for strata with little coverage, cases that are too easy, disagreements between judges and experts, and thresholds that no longer match the business cost of error. Do not respond to every regression by loosening a gate. First identify whether the case exposes a valid product requirement, stale annotation, provider variance, or a genuine defect.

Teams that need help establishing the first contracts, instrumenting a RAG pipeline, or connecting a gateway to release policy can use the site’s AI architecture and engineering service paths. The useful engagement is not a generic model assessment; it is a working evaluation loop that your team can operate after delivery.

Frequently asked questions

A release decision people can trust

The goal of an evaluation harness is not to manufacture a single quality number. It is to make an AI release explainable: this exact pipeline, against this versioned evidence, preserved these safety boundaries, improved these user tasks, stayed within this cost and latency envelope, and can be rolled back to this baseline.

That is the difference between adding an AI feature and operating an AI system. Golden sets make requirements concrete. Stage metrics make failures diagnosable. CI gates make changes reviewable. Shadow traffic makes offline confidence accountable to reality. Gateway telemetry makes quality economics visible. Together, they give platform teams a release process that survives production rather than a demo that survives only applause.

Need a production evaluation loop?

If you need contracts, instrumentation, and release gates for RAG or agents on your stack, see the AI implementation service.