This is a copybook, not four foreign notebooks

Projects often hold four data families. For SFT, do not confuse what teaches with what measures — or you remix the gauge from the series overview with the batch again.

SFT is the copybook: given this input, we want one output (or a refusal). Preferences compare two essays. The CI golden is the exam key. The buffer is desk drafts still unscored.

The corpus may be wider and noisier than golden: the goal is to learn a pattern, not lock a regression gate. If a record cannot be stated as “on this input we want that output,” it is not yet an SFT exercise.

Rules for the line before the stack of pages

Before labeling, put the pair contract in one place (meta.yaml or a short RFC) — like the model letter at the top of a copybook:

Without that, two experts write two “correct” answers of different length, and the model learns the average of noise. If search plus a hard template already closes the contract, check when fine-tuning is justified first. SFT earns its keep when stable behavior (format, refusal, tone, jargon) will not hold on prompt and retrieval alone.

One cell, one set of fields

A minimal instruction-tuning record is not a wall of text — it is fields that survive checkpoint and library changes:

{
  "id": "sft-reg-0142",
  "messages": [
    {"role": "system", "content": "…one system-prompt version…"},
    {"role": "user", "content": "May PPE be stored without an issue log?"},
    {"role": "assistant", "content": "Per policy v3.2 §4.1 — no. …"}
  ],
  "meta": {
    "domain": "ot_pb",
    "intent": "policy_qa",
    "doc_version": "3.2",
    "language": "en",
    "refusal": false,
    "source": "editor",
    "difficulty": "medium"
  }
}

Four rules save weeks:

1. One system-prompt version per corpus manifest (or an explicit system_version). Five different systems in one JSONL teach policy conflict.
2. Stable id across corpus versions so an answer edit is not a silent new case.
3. Slice metadata: domain, intent, document version, language, refusal, source. Without them you cannot say which slice failed.
4. Chat template matches production. If production has no system, do not train on a long system that will never be sent.

Classification may use a simpler input / label shape; the principle stays: schema + metadata + manifest version.

Where the lines in the book come from

Four sources, each with its risk.

A mix that works in pilots: 30–40% core written or approved by experts; periphery grown with synthetic paraphrases by intent template; tail from the failure buffer after review. Photocopies without quota and rubric fill the book with confident average — the model turns polite and empty.

“Model generated a pair” ≠ “pair verified.” In high-cost domains, final status needs a human. Synthetic data does not sign the manifest.

Not line count — a map of exercises

“How many examples?” without a coverage map is empty. Working ranges (not dogma):

“More” helps only when intent and condition coverage grow — not paraphrases of one line. Map intents × conditions (source present / absent; current / obsolete version; one doc / conflict). An empty cell beats a hundred similar “success” answers.

If one editor wrote every user line in literary prose, the model fails on conversational production. Mix fragments, typos (if they exist in traffic), and formal mail — in traffic shares, not annotator taste.

Narrow classification on a small model sometimes needs only hundreds of examples — see the Qwen 0.6B case. Open policy dialogue without refusals will not be saved by ten thousand pairs.

Exercises where the right answer is “do not write”

A corpus without refusals teaches the model to always answer. In an enterprise assistant that is a straight road to invented clauses.

Minimum “negative” skills in the copybook:

1. Refuse without a source — “not in the provided chunks; I will not invent.”
2. Refuse on ACL — no rights; do not paraphrase from “general knowledge.”
3. Refuse on obsolete version — document withdrawn.
4. Escalate — two norms conflict; hand off by template.
5. Out-of-scope refuse — weather, jokes, if the product forbids them.

Such cases often take 15–30% at pilot start — otherwise the positive class crushes behavior. Presence of every refusal type from the contract matters more than the exact percent.

“Similar but wrong” answers usually do not belong as assistant — that is preference / DPO territory. For SFT, prefer a strictly correct output or a record whose correct output is a refusal. Choosing between two phrasings needs a separate family, or one notebook pulls the model both ways.

Do not open the lesson until the notebook passes inspection

Training does not start from “a folder on disk.” Gate first — like checking the copybook before the exam:

flowchart TB
  sources[Sources: editor / prod / synthetic] --> normalize[Normalize schema and template]
  normalize --> lint[Static checks]
  lint --> split[Train / dev / holdout]
  split --> trainJob[SFT job]
  split --> evalGate[Eval on golden]
  evalGate --> release{Regression gate}
  release -->|pass| ship[Ship adapter]
  release -->|fail| buffer[Corpus fix buffer]
  buffer --> sources

What catches pain before GPU:

• schema: required fields, types, allowed role;
• one system per manifest (or explicit versions);
• dedup on text and near-duplicates;
• zero overlap with eval golden;
• refusal share in a corridor;
• length, empty content, giant log pastes;
• PII and secrets by policy;
• language label matches audited text.

The gate’s output is a manifest: version, hash, slices, approver, rubric link. Without it, “model A vs B” also compares different notebooks.

Homework must not match the exam

A common trap: take the last 10% of the file as test. If the file is sorted by time or author, the test does not look like production. Worse: edit train answers while peeking at the same phrasings that go into the customer report.

Practical rules:

1. Holdout — tens to hundreds of cases closed during prompt and corpus iteration. “Ready for pilot” reports only from holdout or a separate golden.
2. Visible dev — for debugging; do not sell its metrics to the business as final.
3. Eval golden versioned apart from the train manifest. Cross-links are fine (“train v0.4 scored on gold v0.3”); embedding golden inside train JSONL “for convenience” is not.
4. Paraphrase is leakage too. “May PPE be stored without an issue log?” and “Is an issue log mandatory for PPE?” must not split across train and holdout without a policy. Search near-duplicates.

RAG eval goldens are built differently — golden sets for RAG. RAG checks retrieval and grounding; SFT teaches phrasing and answer policy. Shared questions across loops are leakage candidates.

Enterprise example: a policy assistant

An internal assistant answers on current policies. Retrieval already returns chunks; answers still drift in format and invent clauses missing from the chunks.

Contract: answer only from provided chunks; require doc_id and version; otherwise refuse; formal tone.

Corpus v0.2 (example mix):

Synthetic only for paraphrases of success and “not in chunks,” 25% quota, mandatory review. After the pilot, “confident answer on empty retrieval” failures became refusals in v0.3 — not new successful fantasies.

Adapter release metric: not average “looks like golden,” but contract-violation rate on holdout below a threshold.

Industrial example: operator instructions

A shop-floor kiosk or tablet: the model helps find the next instruction step and allowed tolerances. Errors cost more than an office chat.

Copybook differences: short lines, speech-recognition noise, mixed language (local + equipment codes); hard refuse on “how do I bypass the interlock” and off-clearance asks; metadata — line, machine, tech-card version, clearance level.

Here 800–1,500 approved pairs with full refusal and line coverage beat five thousand synthetic polite dialogues. Same logic as the handwritten-digit OCR breakdown: resemblance to operations first, scale second.

Nearby reading

A narrow “classify → label” experiment shows a small model with clean labels can beat a large dirty checkpoint. Do not copy that win onto open dialogue one-to-one: the unit is answer policy, and “almost right” text costs more.

This piece answers: how to design the corpus once “we fine-tune” is decided.

Where teams ruin the copybook

Chat dump without a schema. Many systems, no id — zero reproducibility.

Zero refusals. The model learns to always be helpful; production gets confident hallucinations.

90% synthetic without review. Pretty loss curves, empty pilot.

Train on the eval golden. Metrics cheer until the first unfamiliar ask.

One author for every user line. Gap to employee or operator language.

SFT and preferences in one file. The model cannot tell “one correct answer” from “what people like more.”

No manifest. You cannot roll back the corpus or explain a regression.

Ignoring ACL and PII in logs. Legal and security debt in the same JSONL.

What to try today

1. Write a one-page pair contract: input, output, refuse, escalate, tone.
2. Draft a coverage map of intents × conditions and mark empty cells — that is the labeling queue, not “another LoRA.”
3. Add at least five refusals of each type from the contract.
4. Run the static gate before the first training job.
5. Name a manifest owner — without whose sign-off the file never enters training.

Finish the copybook before spinning the adapter again. Otherwise the next metric bump is just copying from the same notebook.

FAQ

Next in the series

• Dataset engineering overview — families and lifecycle.
• JSONL, shards, and the path to the GPU — physical layout.
• Golden editors — rubric and adjudication (golden-editors-annotation-workflow-2026).
• Train / eval / holdout and leakage (dataset-splits-leakage-2026).
• Preference data: DPO and pairs (preference-dataset-rlhf-2026).
• Synthetic and hybrid pipelines (synthetic-human-dataset-pipeline-2026).