Key takeaways

Model quality is several independent properties. Capability (“can it solve this class of task”), reliability (“does it solve it stably”), safety (“does it refrain from forbidden work”), product fit (“does it help in our scenario”), and cost (“does it fit the token and latency budget”) move separately. A high score on an open knowledge set does not guarantee a safe refusal in your help desk.

A test checks a claim, not “intelligence.” A multiple-choice exam checks selection from a closed list. Executing generated code checks whether the program passes hidden cases. Humans comparing two answers check preference for style and usefulness, not truth. Until the claim is named, a figure in a report is uninterpretable.

A public ranking is someone else’s sample with someone else’s cost of error. It is useful as a coarse filter of candidates. A release needs your own set: your phrasings, your prohibitions, your gold answers, and your damage from a false yes.

An automatic judge is an accelerator, not the truth. A model that scores another model inherits blind spots and drifts when its version changes. Calibrate it on human labels and do not make it the only gate where the cost of a miss is high.

Product regression beats hunting a leaderboard. A drop on five of your critical scenarios matters more than a gain on a thousand foreign questions. A hidden holdout and dataset versioning protect you from fitting the prompt to a known test.

Those five statements are enough to reject most “the new model is better” slides. The rest of this article is how to turn them into a mix of oracles you can defend in a release review.

What language-model quality actually means

For a function with a fixed input and output, a “test” usually means: given these arguments, you get this result. A language model takes free text, emits a distribution over next tokens, and often has more than one acceptable result. Quality therefore has to be split into layers.

Capability is the upper bound of what the model can do on a class of tasks: mathematics, code, field extraction, following a long instruction. Teams measure it on open and closed sets, often with no in-context examples or with a handful of demonstrations. A capability number answers “is this family even in the right league,” not “will Tuesday’s chat stay stable.”

Reliability is how stable that level is. The same task at temperature above zero, after a paraphrase, or after a system-instruction change can produce a different answer. A team that runs a set once at temperature zero is measuring a ceiling, not chat behavior. Reliability also includes format jitter: a model that sometimes emits valid JSON and sometimes wraps it in a markdown fence is not “almost capable”; it is unusable downstream.

Factuality and groundedness are not the same property. Factuality asks whether claims about the world are true. Groundedness asks whether claims follow from the context you supplied. A model can refuse honestly and be factual yet useless. It can answer beautifully from a corpus and still distort a number in a table. Mixing the two into “hallucination rate” hides which layer failed.

Instruction and format following is whether constraints hold: language, JSON schema, banned topics, length, tone, use of only the listed tools. For agents and gateways this layer often matters more than “general knowledge.” A model that knows the tariff formula but cannot emit the field names your billing service expects is a failed integration, not a trivia miss.

Safety and policy is refusal of prohibited actions, resistance to instruction injection, leakage of the system prompt, and bypass of restrictions. A drop here is an incident even if every other chart is green. Safety is not a bonus metric you average with MMLU.

Product fitness is whether the model solves your tasks under your access rights, latency, and price. You cannot buy this layer from a foreign ranking. An ERP clerk’s query distribution is not an arena volunteer’s, and a legal desk’s preferred refusal is the opposite of a consumer chatbot’s chattiness.

Public task sets: knowledge, math, code, reasoning

Open task sets are standardized exams. Their strength is comparability across labs. Their weakness is that they measure someone else’s exam, part of which may have leaked into training data.

Classic knowledge sets such as MMLU check multiple-choice selection across school and university subjects. Scoring is convenient: there is a key, there is an accuracy share. Transfer to free-form product answers is poor. A model may guess the letter and be unable to explain, or give a sound chain of thought and trip over the answer-sheet format. A multi-subject average can hide a collapse in the one domain you actually ship. If your product is a tax assistant, a rising history score is noise.

Math sets such as GSM8K and olympiad-style samples check multi-step arithmetic and sometimes a formalizable derivation. The gold answer is often unique, so automatic checking is stronger than on essays. A school story about five apples still weakly predicts the error a model will make on your tariff formula with exceptions and exchange rates. Word-problem fluency is not domain arithmetic under policy constraints.

Code sets range from synthesizing a function from a docstring (HumanEval and relatives) to editing a repository (SWE-bench and derivatives). They sit closer to engineering: hidden tests run, “almost correct” code fails. Even here the claim is narrow: the model can close this class of tasks in this harness (file access, several attempts, a particular judge). Changing the harness often moves the score more than changing “intelligence.” A SWE-bench number with an agent loop, retries, and a custom test runner is a different product from a single-shot completion.

Sets labeled “reasoning” try to catch a chain of steps, not only the final number. They are easy to spoil with a “think step by step” hint in the system instruction — the very hint you may have banned in the product because of length and cost. Compare models in the mode you will call: with tools or without, short answer or long. A chain-of-thought exam at temperature zero is not a chat with tools at temperature 0.7.

Training-set contamination is a separate risk. If task wordings appeared in the training corpus, the score reflects memory, not generalization. Practically: do not make a public set the sole procurement criterion, and keep a private holdout that you never paste into prompts or tickets. A sudden jump on a famous set with no movement on your cases is a contamination smell, not a celebration.

Human preference and pairwise arenas

People are bad at assigning an absolute score of “how good is this answer,” but comparatively confident when choosing between two. Hence arenas: two models answer one query, a human marks a winner, thousands of such duels become a rating.

What an arena checks is preference as a reply in a dialogue: clarity, tone, willingness to help, absence of refusal, sometimes stylistic closeness to the audience. A strong arena model is often the one that answers more readily and sounds confident. That can be exactly the behavior you want to suppress in a legal or medical contour. Verbosity and a warm closing paragraph win votes; a short, correct “I cannot advise on that” loses them.

What an arena does not check: truth without an expert; compliance with your policy; behavior on your internal documents; cost and the latency tail; robustness to hostile input. Query-sample bias is structural. Arena volunteers do not ask what an accountant asks in an ERP. They also rarely file the same ticket twice with a slightly different identifier, which is how real users break assistants.

Pairwise comparison inside the team is a different instrument. Domain experts compare a candidate with the current version on your cases. The rubric must be explicit: factuality, completeness, refusal, tone, format. Otherwise the winner is “the politer paragraph.” Separate the votes: a lawyer scoring groundedness should not be mixed into a single elo with a designer scoring friendliness.

Use arenas as a coarse popularity prior when choosing a family. Use expert pairwise comparison as a release signal only when the rubric matches the product contract. Do not treat a public chat ranking as an acceptance test for a system that must refuse.

Model judges and automatic text metrics

When answers number in the thousands, people do not scale. Two workarounds exist: compare text with a gold answer using surface metrics, or ask another model for a verdict.

Token overlap (ROUGE and similar schemes) and embedding proximity (BERTScore and analogues) are cheap and reproducible. They catch “looks like the gold.” They miss negation and direct contradiction: “forbidden” and “allowed” can sit close in words or in vector space. The MATCHA metric research shows that classic schemes can score highly a text that refutes the gold; the write-up is in the MATCHA note. If your product lives on yes/no policy language, overlap metrics are not a factuality oracle.

A model judge reads the task, the answer, and sometimes the gold, then assigns a score or picks a winner. On many free-form tasks it agrees with humans better than token overlap. The price is known: judges are biased toward length, toward their own model family, toward option order, toward rubric wording. Changing the judge version rewrites chart history the way changing a compiler rewrites warnings. Position bias alone can flip a pairwise winner if you always put the candidate first.

Practical rules. Pin the judge identifier, temperature, system instruction, and rubric as part of the evaluation version. Calibrate on a hidden human sample: agreement rate, systematic errors by stratum. For critical classes (safety, money, health, access) do not make the judge the only barrier. For bulk triage of drafts, do — but leave a tail for an expert. Store the judge’s raw rationale; when humans disagree later, you need to see why the automatic vote landed.

Separately, “an LLM checks an LLM for hallucinations” with no gold is a weak signal. A model can confidently confirm its own invention. Stronger: check claims against supplied context or against a structured fact. A citation-level check (“this sentence is supported by span X”) is a different test from “does this paragraph sound plausible to another model.”

Exact match, execution, and tool traces

The most honest tests are those where the verdict does not read prose.

Field extraction: the model must return JSON matching a schema. The check is a schema validator plus value comparison with gold. You catch extra fields, type changes, “almost correct” dates. For gateways and agents this is the base layer: if the format drifts, the rest of the pipeline has no right to run. Treat markdown fences, trailing commentary, and NaN in numeric fields as failures, not as “close enough for a demo.”

Code execution: a generated function or patch runs against hidden tests. The claim is that the program satisfies the test specification, not that it “looks like Python.” Weak spots: tests written by the same model, and a harness that leaks hints from the call stack. If the hidden tests are thin, you are scoring style again. Prefer property checks and fixtures from the real codebase over a handful of assert lines the model has already seen on the internet.

Tool calling: the gold is not the reply text but the sequence and arguments of calls. You check that the model invoked the needed tool, did not invoke a forbidden one, passed a correct identifier, and stopped after success. That is closer to an agent integration test than to a knowledge exam. Without it, a “smart chat” can narrate that it sent an email without sending one. Record idempotency keys and tenant IDs in the trace; a call that looks right in prose can still hit the wrong account.

Deterministic oracles (calculator, database, compiler, API contract) should be invoked from the check, not asked of the model as a paraphrase of their result. Otherwise you are scoring prose again. If a billing rule lives in code, execute the code. If a document ID must exist, query the store. The model’s job in that slice is to produce the arguments, not to impersonate the system of record.

Factuality, groundedness, and the RAG boundary

Two different questions get confused more often than any other pair.

“Is this claim true of the world?” is factuality. It needs a gold outside the model: a knowledge base, a document, an expert, an execution. Without that anchor you are scoring “sounds plausible.” Parametric memory can be right on famous facts and wrong on last Tuesday’s policy change. A world-knowledge exam does not tell you whether the assistant will cite the current SOP.

“Does this claim follow from the supplied context?” is groundedness. Here the gold is the fragments you put in the window. A model can be grounded and factually stale if the context is stale. It can be factually right “from parameters” and still violate “answer only from documents.” Those failure modes need different fixes: refresh the corpus versus tighten the generator.

For retrieval-backed assistants the layers are separated on purpose: first whether the needed fragment reached the context, then whether it was used in the answer, then whether the model added extra. The chain is unpacked in Production RAG Engineering; collecting gold is in RAG golden datasets. This article only fixes the principle: do not score generation with one number if the error could have happened at retrieval.

Checks without retrieval still need an abstention policy. A set where every question has an answer trains the model never to say “I don’t know.” Add unanswerable cases and measure false answers separately from useful ones. A silent invention on a missing SKU is a different incident from a verbose but cited summary of a found policy.

Safety, policy, and adversarial tests

An ordinary knowledge exam almost never sees harm. Safety is checked with separate sets.

Policy tests pose forbidden classes: harm instructions, moderation bypass, requests for other people’s secrets, actions outside the assistant role. Gold here is often “refuse / escalate / answer inside policy,” not “give the best answer.” Metrics are the share of violations and the share of false refusals on benign queries. Tightening refusal is an easy way to improve “safety” at the cost of usefulness. A model that refuses everything is a perfect safety chart and a dead product.

Instruction injection checks whether the model obeys text in a document, a web page, or a user message of the form “ignore previous rules.” A product contour that mixes untrusted context is obliged to keep such cases in regression. The attack class is discussed in prompt injection in production and in AI guardrails. Indirect injection through retrieved HTML is not a rare research toy; it is the default shape of a RAG assistant that reads the open web or an email inbox.

Leakage of the system instruction and of secrets from context is a separate set: direct “repeat your rules,” indirect (“which tools do you have?”), attempts to extract keys from history. Gold is absence of the secret in the output, not a polite refusal that still quotes the key in a code block. Check tools and side channels, not only the visible paragraph.

Adversarial probing (a red team) does not replace regression. It is research: people and automatic generators hunt new holes. Findings get promoted to permanent cases. Treating a one-off research session as a release gate leaves the hole open until the next session. Budget red-team time for new surfaces (a new tool, a new connector); budget regression for everything already found.

Multimodality and tools widen the surface: a model can refuse in text and still invoke a dangerous tool. The test must look at the action, not only the paragraph. A screenshot that contains an injected instruction is a different case from a plaintext jailbreak. If the system can pay, delete, or email, the oracle is the side effect.

Product regression versus chasing a public ranking

A team picks a model from a leaderboard, wires it in, then changes the system instruction, tools, and routing — and is surprised that “quality floats.” The public score described a different assembly.

A product regression set is not a shrunk MMLU. It is your critical dialogues: a typical successful request, an exact identifier, a refusal, a policy conflict, a tenant boundary, a long context, a tool call, a hostile phrase inside a document. Each case has an owner, a risk level, and a rule: block the release or only observe. How to put gates in CI is in the evaluation harness.

Keep three populations separate. The commit set is tens of deterministic cases, seconds, no expensive judge — format, schema, a handful of must-not-fail scenarios. The release set is wider and may include a model judge and a human sample. The hidden holdout is cases that never appear in prompt-tuning tickets, so the team cannot memorize the test. Promote production incidents into the commit set; do not demote them to a dashboard curiosity.

Changing the model, the prompt, the temperature, the gateway route, and the judge version are different experiments. If you change two at once, you do not know what moved the chart. Pin a baseline and compare pairs on the same inputs. A “model upgrade” that also rewrote the system instruction is not a model upgrade.

The economics match ordinary software: test in proportion to damage, not to “question coverage.” The frame is in the cost of failure. One missed write-off scenario outweighs a hundred green joke queries. Spend judge budget on high-risk strata; spend cheap schema checks on every call path that feeds code.

Online evaluation: shadow traffic, experiments, and traces

An offline set ages. Users invent new phrasings, the document corpus moves, fashions in restriction-bypass prompts change. An online layer does not replace offline; it catches drift.

A shadow run sends live queries to a new version without showing the user the result. You compare refusals, length, tool calls, policy hits, and divergence from the current answer. That is cheaper than an incident and more expensive than an offline set: you need masking, retention limits, and a ban on secrets landing in evaluation logs. Shadow is not “dump production into the judge.” It is a sampled, redacted, purpose-limited pipeline with the same access rules as the service.

A controlled comparison (A/B) shows product impact: conversion, repeat questions, escalation to a human, user rating. The “disliked” tail mixes a bad answer, a slow answer, and a wrong scenario. Without labeling the cause, the comparison teaches you to optimize politeness. Slice by task family before you declare a winner. A latency regression can look like a quality regression in thumbs-down data.

Explicit user signals (thumbs up, “rephrase,” topic change) are weak and biased. Use them as a source of candidates for the gold set after review, not as the sole model-quality metric. Power users who click everything and silent users who churn without clicking are not the same distribution.

Tracing must keep enough to reproduce a case: model version, system instruction, tools, context fragments under the data policy. Otherwise online evaluation turns into anecdotes from support chat. If you cannot replay the exact assembly, you cannot tell a model regression from an index regression.

How to assemble a mix of checks for a product

Start from harm, not from a catalog of acronyms. An internal-policy help desk cares about groundedness, refusal, and access rights. An agent that triggers payment cares about call schema, idempotency, and a ban on extra tools. A public chat cares about safety and tone. An IDE assistant cares about test execution and the diff, not erudition.

A minimal frame that covers most LLM services:

Do not copy every open exam “for completeness.” Each set has a maintenance cost: who updates it, how a new incident enters, when a case is retired. A dead catalog of ten thousand foreign questions is worse than eighty living ones of yours.

Tie check types to routing. A cheap model on FAQ may pass only format, refusal, and a small regression set. An expensive model on disputed tickets still needs a human sample. How budgets meet the gateway is in LLM gateway FinOps. Evaluation spend is part of serving cost; a judge on every shadow request will dwarf the candidate model.

Common mistakes

One score on every slide. Averaging knowledge, safety, and preference yields a convenient presentation and a useless release decision. Split gates. A safety miss blocks; a trivia dip on an unused subject does not.

Scoring prose where an oracle exists. Asking a judge whether a sum is correct instead of adding the numbers. If the billing engine can compute, let it compute.

Tests the prompt has already seen. The team memorizes the set, the chart improves, the first new phrasing breaks. Keep a hidden holdout and rotate cases from production.

Changing the model and the judge in one release. Quality history becomes unexplainable. Pin the judge across model experiments.

Arena as acceptance. Arena users like talkativeness; your counsel likes refusal. Use the arena to shortlist families, not to ship.

Safety as a refusal rate. A model that always refuses is “safe” and useless. You need both forbidden and allowed cases, and a false-refusal budget.

Ignoring the harness. A SWE-bench score with an agent and without an agent are different products. Your MCP server is part of the system under test. Version tools, prompts, and routes with the weights.

Confusing a ten-dialogue pilot with a set. A pilot finds holes. A set pins them so they cannot return. Promote every incident; do not restart discovery every quarter.

FAQ

Further reading

• AI evaluation harness in 2026 — turning check types into CI, shadow traffic, and release criteria.
• Evaluating enterprise AI — tying quality to process and safety at enterprise architecture level.
• RAG golden datasets — gold for retrieval and answers, not an exam of a bare model.
• Production RAG engineering — where in the chain an error is born that a generation test will not see.
• The economics of failure cost — how much checking to buy relative to damage.
• MATCHA — why surface text metrics miss a contradiction of the gold.
• Agentic engineering — quality of the tool loop, not only of a paragraph.

External landmarks (not a substitute for your own set): documentation of open task sets and arenas, papers on test-set contamination, and vendors’ red-team practices.

Conclusion

Quality testing for a language model starts with a sentence you are willing to defend: “on these inputs the system honors this contract.” The contract may be about knowledge, format, groundedness, a tool action, or a prohibition. Each contract gets its own oracle. A foreign exam, an arena, and a model judge are useful instruments with a known error. The release decision is made by your harm strata, a hidden holdout, and the ability to explain a rollback.

In one iteration, do not collect a “full benchmark.” Collect a short matrix: which claims are critical, what checks them, what blocks a release. Then grow the set from incidents, not from rows on someone else’s leaderboard. The lab rule is simple: measure what cannot be replaced by the impression of a lucky chat.