An export is not inspection

Dumping CSV or buying labels on a marketplace is like pouring parts into a crate and calling it quality control. Dataset engineering is a repeatable process with artifacts — without them, acceptance does not work:

No schema, golden owner, and version — you are still experimenting with prompts. Fine on day zero. Dangerous once a pilot is promised to the business: the gauge must exist before the “we improved” report.

A new machine does not fix a bent gauge

In enterprise and industrial work, error cost is asymmetric. A wrong class on a demo image is annoying. A wrong reading of a safety clause is an incident. Buying a “smarter” model is like replacing a machine tool: if the reference is bent, the new machine stamps the same bend faster.

So “which model?” belongs after “which input/output contract can we verify on data that looks like production?”

Three signals you should fix the gauge, not the checkpoint:

1. The metric jumps without reproducibility — one pull request changed prompt, set, and model. The “+12%” cannot be tied to one change.
2. Good test scores, bad pilot — training does not match the real stream. Classic case: MNIST versus field forms in the handwritten-digit OCR breakdown.
3. No labeling owner — “correct” is whoever last looked at the chat.

A bad set costs more than it looks: overfitting to eval, GPU hours on noise instead of fixing retrieval, no way to roll back “set v3,” experts who stop reviewing. Dataset engineering makes improvements explainable: version, slice, gate — artifacts both engineers and product owners understand.

Agree what “good” means first

Before labeling, put the contract in one document (or the set’s meta.yaml) — like a drawing tolerance before the first batch:

Without that, two experts label the same case differently, and low inter-annotator agreement is missing rules — not a “hard task.” Broader pre-production eval thinking is in evaluating LLMs before production; here the focus is the data that feeds that loop.

Four crates you must not mix

One warehouse can hold four crates. Dump them into one and you get a pretty metric and a false release.

The training corpus teaches how to answer: tone, format, domain patterns. For SFT that is instruction→answer pairs. It can be large and noisy at the edges, but it still needs static checks: dedup, ACL scan, no obvious overlap with eval.

Preference data matters when you must choose between two plausible answers: shorter or fuller, with a cite or without. Typical formats are DPO and RLHF-style pairs. Volume is usually smaller than SFT; labeler quality matters more.

The golden set is fixed and versioned, often smaller than training. Each golden record is a reference sample. For RAG specifically: golden datasets for eval.

The mining buffer is raw complaints and errors. It is not a reportable metric until an editor promotes a record into golden or training. Review SLA depends on risk: security incidents in hours, cosmetics in weeks.

One sample, one contract

In DeepEval / Confident AI ecosystems, a golden record is a test-case predecessor: fields are fixed before the run; actual output, retrieval context, and tool calls appear at eval time. In your own repo, a minimal JSONL contract looks like this:

{
  "id": "reg-2026-0142",
  "input": "What is the validity period of the working-at-height instruction?",
  "expected_output": "Answer citing the in-force edition И-ОТ-12 dated 2024-03-01; if the corpus only has a revoked version — refuse and name the version.",
  "acceptance": "citation_required",
  "metadata": {
    "domain": "industrial_safety",
    "risk_class": "high",
    "language": "ru",
    "source_ids": ["И-ОТ-12"],
    "tags": ["height_work", "validity"]
  },
  "annotator": "editor:petrov",
  "rubric_version": "ot-rubric-1.2",
  "dataset_version": "golden-regulations-v3"
}

acceptance and rubric_version tie the sample to the rules. Without them, six months later nobody can explain why “good” quietly changed — the gauge was swapped in silence.

Who owns the meaning of “gold”

The term golden editor comes from LLM eval tools: a UI where an expert fills fields and assigns review. On the floor, the job is closer to a metrologist who owns the gauge than to a button in a product.

Engineers own schema, validation, and CI. Golden editors own the meaning of gold. Without that split you get dataset theater: files exist, trust does not.

Detail lives in golden-editors-annotation-workflow-2026.

From the line back to the inspection bench

The loop is alive: production creates new failures, documents and rubric-guided synthetic feed training, people at the reference decide what enters a version.

flowchart TB
  subgraph sources [Sources]
    prod[Traffic and logs]
    docs[Documents and policies]
    synth[Synthetic by rubric]
  end

  subgraph human [People]
    editors[Golden editors]
    adjud[Adjudication]
  end

  subgraph artifacts [Artifacts]
    buffer[Mining buffer]
    golden[Golden set vN]
    train[Training corpus]
    pref[Preference pairs]
  end

  subgraph gates [Control]
    eval[Eval harness]
    ci[Regression gate]
  end

  prod --> buffer
  docs --> train
  synth --> train
  buffer --> editors
  editors --> adjud
  adjud --> golden
  adjud --> train
  train --> eval
  golden --> eval
  eval --> ci
  ci -->|pass| release[Model / prompt release]
  prod -->|new failure| buffer

Buffer is not golden until rubric and review exist. Every run logs dataset, model, and prompt versions. Holdout stays closed during prompt iteration — otherwise people file the ruler. One experiment, one variable: do not change parser, prompt, and set in the same release.

Teach, measure, and hide the ruler

Three evaluation pools get confused most often. Training is the fine-tuning corpus. Dev is the visible set the team iterates on. Holdout is a rare pre-release check you must not “fit by eye.”

flowchart LR
  subgraph pools [Data pools]
    train[Train / SFT]
    dev[Dev — visible eval]
    hold[Holdout — hidden]
    mine[Mining buffer]
  end

  train -->|train| model[Model / prompt]
  dev -->|iterate| model
  dev -->|CI metrics| gate[Gate]
  hold -->|rare check| gate
  mine -->|after editors| dev
  mine -->|after editors| hold

Leakage is when gauge and batch share the same steel: paraphrases across train and eval, few-shots from golden, fine-tuning on text near eval. Before trusting a score, search near-duplicates across pools. Separate article: dataset-splits-leakage-2026.

When retrieval is enough — and when it is not

Not every bad answer needs fine-tuning. Sometimes you need to find the right document and not invent.

Criteria live in when fine-tuning is needed: without a business reason, an SFT corpus is an expensive way to memorize noise.

Volume depends on the task, not the fashion

There is no universal “ten thousand examples” — different tasks have different acceptance units.

Hard negatives matter especially: without them the model learns easy cases and fails on neighboring policy sections. Expected-refusal cases are not labeling mistakes — they are a requirement. For a policy assistant, 15–25% of golden with refuse or escalate is reasonable.

Domain changes what is scarce

Full treatment: dataset-parameters-by-domain-2026. Short map: each domain lacks a different resource — citations, tolerances, API freshness, or channel quality.

Who approves the gauge

Write down who approves promoting a buffer case into golden and who may change the rubric without a major set version. Without owners the gauge drifts quietly — and everyone argues about the model.

Without a version number, the metric is a different ruler

Pin every eval release to a manifest — otherwise you compare September to August with different gauges:

dataset_id: golden-regulations
version: v3.2.0
schema_version: 2
rubric_version: ot-rubric-1.2
content_hash: sha256:...
record_count: 142
holdout_count: 28
created_at: 2026-08-15
changelog: "Added 12 obsolete-edition cases; fixed id reg-0088"

In CI: schema validation, unique id, ban empty expected output without a refusal acceptance, check holdout is not in training. dataset-versioning-governance-2026 expands governance.

Measure the quality of the set itself

Model accuracy is not the only scale. Look at the “set” artifact:

Enterprise metrics live in evaluating enterprise AI; here — the quality of the gauge, not only the machine.

Enterprise example: a policy assistant

An industrial holding launches an assistant over internal policies.

Week 0 — contract. Product and HSE lock: answer only with a citation; on version conflict — refuse; off-HSE questions — refuse with a route to HR/IT.

Weeks 1–2 — golden v0 (80 records). Two HSE engineers and a methodologist label by rubric. Visible eval is team-only. Holdout: 20 records with the tech lead.

Weeks 3–4 — RAG baseline. Indexing follows data preparation. On golden v0 the “obsolete version” slice fails 40%. Conclusion: not the model — an index without validity dates and missing refusals in the prompt.

Weeks 5–8 — SFT corpus (600 pairs). Only after retrieval is fixed. Question→answer-with-clause pairs. Training does not overlap holdout.

Ongoing — buffer. Two finalized goldens per week from the pilot. Quarterly rubric review and a major set version.

A quarter later the report is reproducible: golden version, index manifest, model — and you can see what lifted the “obsolete policy” slice. The gauge stayed the same tool; what you measured with it changed.

Industrial example: field forms and OCR

In the handwritten-digit OCR breakdown, the environment sets the parameters: not a 28×28 glyph but a perspective photo; not one digit but a sequence in a cell; labels from a database, not an army of annotators. The same law for language models: channel, noise, and task setup in the set matter more than checkpoint size. The gauge must reflect the shop floor, not the lab bench.

Link to the RAG cluster and the book

This series does not replace the RAG cluster: it covers training, labeling, golden editors, and data governance where an index alone is not enough.

Where teams break the ruler

One file for train and test — without duplicate and paraphrase audit. Golden without a version — last month’s metrics do not compare. Synthetic instead of experts on rare cases multiplies plausible noise. No refusals in golden — the model learns to guess. Empty metadata — tenant and risk regressions stay invisible. Fine-tuning before fixing retrieval — SFT memorizes retrieval mistakes. Labeling without a rubric — endless fights and low agreement.

What to try today

1. Draw the four families and write where the data sits now.
2. Name a golden-set owner and lock one golden record (JSON above).
3. Run a leakage search across train, visible eval, and reported eval.
4. Add five expected-refusal cases before changing the model.
5. Write manifest v0.1.0 even for thirty records — version habit from day one.

Fix the gauge before buying a new machine: otherwise the next “twelve percent” will be unreproducible again.

FAQ

Next in the series

• Golden records and editors — rubric, adjudication, review queue (golden-editors-annotation-workflow-2026).
• Train / eval / holdout and leakage (dataset-splits-leakage-2026).
• JSONL, shards, and the path to the GPU (jsonl-shards-large-datasets-2026).
• SFT and preference data (sft-dataset-design-2026, preference-dataset-rlhf-2026).
• Parameters by domain (dataset-parameters-by-domain-2026).

Series catalog: docs/ml-datasets/README.md in the project repo.