Key takeaways

A dataset is a versioned product — schema, owner, manifest (hash + version + labeling rules), and change history in git or a data store.

Training and evaluation are separate loops. Training corpora teach behavior; golden sets measure it. Mixing them without leakage audits produces fake metrics.

Task contract before volume and model. Inputs, outputs, and refusals are defined before LoRA and checkpoint size.

A golden record is a per-example contract: input, expected output or acceptance criteria, metadata. A collection is a golden dataset.

Golden editors are domain experts who create and adjudicate gold. Without them, synthetic data yields pretty scores and weak production behavior.

Parameters depend on task and domain: volume, label unit, hard negatives, and quality gates differ for classification, SFT, retrieval, and CV.

The loop does not end at release: production failures enter a mining buffer, pass editor review, then join gold or training.

Averages lie without slices. A good mean can hide failure on a rare, expensive segment — obsolete policy, wrong tenant, poor scan quality.

How dataset engineering differs from “we exported a CSV”

Dataset engineering is not a one-off log dump or a labeling marketplace order. It is a repeatable process with explicit artifacts:

If you lack schema, golden owner, and versioning — you are prompt-experimenting, not doing data engineering yet. Fine for day zero; risky once the pilot is promised to the business.

Why data beats the next model

Error cost is asymmetric. A wrong demo label is annoying. A wrong reading of a safety regulation is an incident. So “which model?” comes after “what contract can we verify on production-like data?”

Three signals to fix the dataset first

1. Metrics jump without reproducibility — prompt, set, and model changed in one PR.
2. Great offline scores, bad pilot — domain gap (see MNIST vs field forms in handwritten digit OCR).
3. No labeling owner — “correct” is whoever looked at the chat last.

Cost of bad data

• Overfitting to eval — metrics cheer until a novel query arrives.
• Expensive fine-tuning — GPU hours on noise instead of fixing retrieval or rubric.
• Nameless regressions — cannot roll back “dataset v3” because there was no version.
• Domain trust loss — experts stop reviewing if their edits are ignored.

Dataset engineering makes improvements explainable: dataset version, slice, gate — artifacts both engineers and product owners understand.

Task contract: input, output, refusal

Before labeling, freeze the contract:

The contract is the basis of the golden editor rubric. Without it, low inter-annotator agreement is a process failure, not “hard task.”

Broader eval mindset: evaluating LLMs before production. Here we focus on data feeding that eval.

Four data families

Up to four families coexist; confusing them causes leakage and false releases.

Training corpus

Teaches how to respond: tone, format, domain patterns. Can be larger and noisier at the edges — but needs lint: dedup, ACL scan, no obvious eval overlap.

Preference data

When you need preference between two plausible answers: shorter vs fuller, formal vs casual, with citation vs without. Smaller than SFT; annotator quality matters more.

Golden set

Fixed, versioned, often smaller than train. Each golden record is a reference for comparison. RAG-eval measures retrieval + generation; classification — slice accuracy; extraction — field F1.

RAG-specific eval: golden dataset for RAG eval.

Mining buffer

Raw stream: logs, complaints, “model was wrong.” Not reporting metrics until an editor promotes a row to gold or train. SLA by risk class: safety incidents — hours; cosmetic — weeks.

Golden record: minimal schema

In DeepEval / Confident AI, a golden is a precursor to a test case: fixed fields before the run; at eval you add actual_output, retrieval context, tool calls.

Minimal contract in your repo (JSONL in git):

{
  "id": "reg-2026-0142",
  "input": "What is the validity period of the work-at-height instruction?",
  "expected_output": "Answer citing active revision I-OT-12 dated 2024-03-01; if only revoked text exists — refuse and name the version.",
  "acceptance": "citation_required",
  "metadata": {
    "domain": "industrial_safety",
    "risk_class": "high",
    "language": "en",
    "source_ids": ["I-OT-12"],
    "tags": ["height_work", "validity"]
  },
  "annotator": "editor:petrov",
  "rubric_version": "ot-rubric-1.2",
  "dataset_version": "golden-regulations-v3"
}

acceptance and rubric_version tie the row to editor rules. Without them, nobody explains why “correct” changed six months later.

Golden editors: a role, not just a UI

Golden editor comes from LLM eval UIs; in industry the role matters:

Engineering owns schema, validation, CI; golden editors own what “gold” means. Otherwise: dataset theater.

Record states (like Confident AI queue vs push):

Workflow deep-dive: golden-editors-annotation-workflow-2026.

Lifecycle: from production back into the set

flowchart TB
  subgraph sources [Sources]
    prod[Traffic and logs]
    docs[Documents and policies]
    synth[Synthetic by rubric]
  end

  subgraph human [People]
    editors[Golden editors]
    adjud[Dispute resolution]
  end

  subgraph artifacts [Artifacts]
    buffer[Mining buffer]
    golden[Golden dataset vN]
    train[Training corpus]
    pref[Preference pairs]
  end

  subgraph gates [Control]
    eval[Eval harness]
    ci[Regression gate]
  end

  prod --> buffer
  docs --> train
  synth --> train
  buffer --> editors
  editors --> adjud
  adjud --> golden
  adjud --> train
  train --> eval
  golden --> eval
  eval --> ci
  ci -->|pass| release[Model / prompt release]
  prod -->|new failure| buffer

Rules:

• Buffer ≠ golden — no rubric and review → not in reporting.
• Pin version to every run — log dataset_version, model_version, prompt_version.
• Holdout closed during prompt iteration.
• One big variable per experiment — do not change parser, prompt, and dataset in one release.

Train, eval, and holdout

flowchart LR
  subgraph pools [Data pools]
    train[Train / SFT]
    dev[Dev eval — visible]
    hold[Holdout — hidden]
    mine[Mining buffer]
  end

  train -->|training| model[Model / prompt]
  dev -->|iterations| model
  dev -->|CI metrics| gate[Gate]
  hold -->|rare check| gate
  mine -->|after editors| dev
  mine -->|after editors| hold

Leakage: paraphrases in train and eval; few-shot from goldens; fine-tune on text near eval. Run near-duplicate search between pools. Article: dataset-splits-leakage-2026.

When train is needed vs RAG and prompt alone

When fine-tuning is justified: when fine-tuning is needed.

Parameters by task type

Hard negatives

Critical for retrieval and classification: “similar but wrong.” Without them, models fail on adjacent policy sections or look-alike SKUs.

Refusals and out-of-scope

Expected refusals are required, not labeling errors. For a policy assistant, 15–25% goldens with refuse/escalate is reasonable.

Parameters by domain (short matrix)

Full article: dataset-parameters-by-domain-2026.

Team and ownership

Write RACI: who approves buffer → golden, who can change rubric without a major dataset version.

Versioning and manifest

Attach a manifest to every eval release:

dataset_id: golden-regulations
version: v3.2.0
schema_version: 2
rubric_version: ot-rubric-1.2
content_hash: sha256:...
record_count: 142
holdout_count: 28
created_at: 2026-08-15
changelog: "Added 12 obsolete-edition cases; fixed reg-0088"

CI: schema validation, unique id, no empty expected_output without acceptance: refusal, holdout not in train. Governance article: dataset-versioning-governance-2026.

Dataset quality metrics (not only model accuracy)

Organizational AI metrics: evaluating enterprise AI. Here: quality of the set artifact.

Enterprise example: policy assistant

Industrial group shipping an internal policy assistant.

Week 0 — contract. Product and safety fix: cite-only answers; version conflicts → refuse; out-of-scope → refuse with route to HR/IT.

Weeks 1–2 — golden v0 (80 rows). Golden editors (two safety engineers + methodologist) label by rubric. Dev eval for the team only. Holdout 20 rows — separate file, tech lead access.

Weeks 3–4 — baseline RAG. Index per RAG data preparation. Eval on golden v0: “obsolete version” slice — 40% failures. Conclusion: index lacks valid_to and prompt lacks refusals — not “wrong model.”

Weeks 5–8 — SFT corpus (600 pairs). Only after retrieval fix. “Question → cited answer” pairs, not raw chunks. Train disjoint from holdout (embedding dedup check).

Ongoing — buffer. Two finalized goldens per week from pilot. Quarterly rubric review and major dataset version.

After a quarter: reproducible report tying golden v1.3, index manifest, and model to slice improvements.

Industrial example: field forms and OCR

Handwritten digit OCR shows environment-driven parameters: not 28×28 glyphs but perspective photos; not one digit but a sequence; labels from DB; different problem statements → different metrics and architectures.

Same law for LLMs: channel, noise, and problem statement in the dataset beat checkpoint size.

Links to the RAG cluster and book chapters

This series does not replace the RAG cluster; it covers training, labeling, golden editors, and data governance where indexing alone is not enough.

Common mistakes

One file for train and test — audit duplicates and paraphrases, not only random_split.

Goldens without versioning — last month’s metrics are incomparable.

Synthetic instead of experts on rare cases — LLMs multiply plausible noise.

No refusals in gold — model learns to guess.

Empty metadata — invisible regressions by tenant, language, risk.

Fine-tune before fixing retrieval — SFT memorizes search errors.

Labeling without rubric — endless disputes, low κ.

What to try today

1. Sketch four families and where data lives today.
2. Assign a golden set owner and freeze one golden record (JSON above).
3. Run leakage search between train/dev and reporting eval.
4. Add five expected-refusal cases before the next model swap.
5. Write manifest v0.1.0 even for 30 rows — version habit from day one.

FAQ

Next in the series

• Golden records and golden editors — golden-editors-annotation-workflow-2026
• Train / eval / holdout and leakage — dataset-splits-leakage-2026
• SFT and preference data — sft-dataset-design-2026, preference-dataset-rlhf-2026
• Parameters by domain — dataset-parameters-by-domain-2026

Series catalog: docs/ml-datasets/README.md in the project repo.