Key takeaways

SFT teaches behavior on input→desired-output pairs. It is not raw-web pretraining and not preference data where “answer A beats B.” Mixing families yields pretty loss curves and strange production behavior.

Contract the pair before you chase volume. Output format, language, when to refuse, when to escalate to a human—fixed before labeling and before synthesis.

Coverage beats size. A thousand similar “short polite answers” lose to a few hundred stratified cases with refusals, edge classes, and real user wording.

Synthetic data drafts; golden editors sign. Domain style and facts are approved by a person with a rubric.

Before training—a static gate: schema, deduplication, zero overlap with the eval set, refusal share, chat-template consistency.

Evaluation must not live in the same file as training. Otherwise you fit prompt and data to the metric instead of testing the system. The broader frame is in evaluating LLMs before production.

How an SFT corpus differs from neighboring families

One project often hosts four families. For SFT it is critical not to confuse their jobs.

An SFT corpus may be wider and noisier than a golden set: the goal is to teach a pattern, not to freeze a regression gate. The Continuous Integration (CI) golden is smaller and stricter. Full taxonomy lives in the series pillar; here we expand only the training loop.

In short: if a row cannot be described as “given this input, we want this output (or refusal),” it is not ready for SFT—it is a note or a mining-buffer candidate.

Start with the instruction → output contract

Before labeling, freeze the contract in one place (dataset meta.yaml or a short RFC):

The contract is the rubric’s backbone. Without it two experts write two “correct” answers of different length and structure, and the model learns the average of noise. Inter-annotator agreement falls not because the task is “hard,” but because rules are missing.

On whether fine-tuning is needed at all: if search plus a hard response template already closes the contract, check when fine-tuning is justified first. SFT pays off when you need stable behavior (format, refusal, tone, domain jargon) that prompting and RAG do not hold reliably.

Record schema and chat template

A minimal instruction-tuning record is not “a blob of text,” but fields that survive checkpoint and trainer library changes.

{
  "id": "sft-reg-0142",
  "messages": [
    {"role": "system", "content": "…one system-instruction version…"},
    {"role": "user", "content": "May PPE be stored without an issue log?"},
    {"role": "assistant", "content": "Per policy v3.2 §4.1 — no. …"}
  ],
  "meta": {
    "domain": "ot_pb",
    "intent": "policy_qa",
    "doc_version": "3.2",
    "language": "en",
    "refusal": false,
    "source": "editor",
    "difficulty": "medium"
  }
}

Rules that save weeks of debugging:

1. One system-instruction version per dataset manifest (or an explicit system_version field). Mixing five prompts in one JSONL trains the model on a policy conflict.
2. id stays stable across corpus versions: editing an answer must not spawn a “new” case with no trail.
3. Metadata for slices: domain, intent, document version, language, refusal, source (editor / synthetic / production). Without them you cannot say which slice failed.
4. Chat template matches production. If production has no system, do not train on a long system that will never be sent at call time.

Classification and extraction may use flatter fields (input / label / spans); the principle is the same: schema + metadata + manifest version.

Sources: experts, production, synthetic, rewrite

Four typical sources—each with its own risk.

A pattern that works in enterprise pilots: 30–40% core written or approved by an expert; periphery expanded with synthetic data on intent templates; tail fed from the failure buffer after review. Synthetic without a quota and rubric fills the corpus with a “confident average”—the model becomes polite and empty.

Do not confuse “a model generated a pair” with “the pair is verified.” In domains with real failure cost (industrial safety, finance, medicine-adjacent) the final row status comes only after a human. The golden-editor role gets a dedicated satellite later; the rule here is enough: synthetic data does not sign the manifest.

Volume, coverage, and diversity

“How many examples?” without a coverage map is meaningless. Practice ranges (not dogma):

“More” helps only when intent and condition coverage grows—not the count of paraphrases of one scenario. A coverage map is intents × conditions (source present / absent; current / withdrawn version; one document / conflict). An empty cell matters more than another hundred similar “success” answers.

User-wording diversity is a separate axis. If every user turn is literary prose from one editor, the model transfers poorly to conversational production. Mix short fragmentary queries, typos (if production has them), and formal mail—in shares that mirror traffic, not the annotator’s taste.

For narrow classification on a small model, hundreds of clean examples can suffice—as in the Qwen 0.6B fine-tuning case. For open policy dialogue without refusals and slices, even ten thousand pairs will not save the pilot.

Refusals, escalations, and “plausible but wrong” answers

A corpus without refusals teaches the model to always answer. In an enterprise assistant that is a direct path to invented policy clauses.

Minimum “negative” skills in SFT:

1. Refusal without a source — “the provided chunks have no answer; I will not invent one.”
2. ACL refusal — no rights to the document; do not paraphrase from “general knowledge.”
3. Withdrawn-version refusal — explicitly state the document is out of force.
4. Escalation — conflict between two norms; hand off to a specialist via template.
5. Out-of-scope refusal — weather, jokes, if the product forbids them.

Such cases often need 15–30% share at pilot start—otherwise the positive class crushes behavior. Exact share depends on domain risk; what matters is every refusal type from the contract, not a magic percentage.

“Plausible but wrong” answers usually do not go into assistant for SFT (that is closer to preference / DPO territory). For SFT you want a hard-correct output and separate rows where the correct output is a refusal. If you need to choose between two acceptable phrasings, open a preference family separately—otherwise one corpus pulls the model in conflicting directions.

Static checks and a pre-training quality gate

Training must not start from “a folder someone dropped on disk.” Minimum gate:

flowchart TB
  sources[Sources: editor / prod / synthetic] --> normalize[Normalize schema and template]
  normalize --> lint[Static checks]
  lint --> split[Train / dev / holdout]
  split --> trainJob[SFT job]
  split --> evalGate[Eval on golden set]
  evalGate --> release{Regression gate}
  release -->|pass| ship[Ship adapter]
  release -->|fail| buffer[Corpus fix buffer]
  buffer --> sources

Checks that catch most pain before the GPU:

• Schema: required fields, types, allowed role values.
• One system per manifest (or explicit versions).
• Deduplication on normalized user text and near-duplicate embeddings.
• Overlap with the eval golden — zero shared id and zero near-duplicate questions.
• Refusal share inside a corridor; empty corridor → alert.
• Length: truncations, empty content, gigantic log pastes.
• PII / secrets: tokens, phones, internal URLs—per project policy.
• Language: language label matches audited text samples.

The gate’s output is a manifest: corpus version, file hashes, per-slice counts, approver, rubric link. Without a manifest, comparing “model A vs B” is meaningless—you are also comparing different data folders.

Separating evaluation: leakage and overfitting

A classic trap: take the “last 10% of the file” as test. If the file is sorted by time or author, the test does not mirror production. Worse—edit train answers while peeking at the same phrasings used in the customer report.

Practical rules:

1. Holdout — tens to hundreds of cases the team does not open while iterating prompt and corpus. “Ready for pilot” reports use only holdout or a separate golden.
2. Visible dev set — for debugging; its metrics are not sold to the business as final.
3. Eval goldens version separately from the train manifest. Cross-references are fine (“train v0.4 evaluated on gold v0.3”); do not stash the golden inside train JSONL “for convenience.”
4. Paraphrase is leakage too. “May PPE be stored without an issue log?” and “Is an PPE issue log mandatory?” must not sit one in train and one in holdout without an explicit policy. Hunt near-duplicates.

RAG evaluation goldens are built differently—see RAG golden datasets. Boundary: a RAG golden checks retrieval and grounding; an SFT corpus teaches phrasing and answer policy. Shared questions across those loops are leakage candidates if one loop peeks into the other.

Enterprise example: policy assistant

Context: an internal assistant answers employee questions on current policies. RAG already returns chunks; answers still drift in format and sometimes invent clauses missing from the chunks.

Contract: answer only from provided chunks; require doc_id and version; refuse if absent; formal tone.

Corpus v0.2 (example mix):

Synthetic data only paraphrased “success” and “not in chunks” rows—25% quota and mandatory review. After two pilot weeks, failures of “confident answer on empty retrieval” entered the buffer as refusals in v0.3—not as new successful fantasies.

Release metric for the adapter: not average “similar to gold,” but contract-violation rate on holdout (invented clause, missing version, answer on empty context) below threshold.

Industrial example: operator instructions

Context: a kiosk or tablet on the shop floor; the model helps an operator find the instruction step and allowed deviations. Mistakes cost more than in an office chat.

Corpus differences:

• Short turns, speech-recognition noise, mixed language (local language + equipment codes).
• Hard refusal on “how to bypass the interlock” and on off-shift / off-clearance requests.
• Rich metadata: line, machine type, tech-card version, clearance level.

Here SFT volume is often smaller than marketing expects: 800–1,500 approved pairs with full refusal and line coverage beat five thousand synthetic “polite” dialogues. The link to real forms and domain gap (lab set ≠ field) is the same logic as in the handwritten-digit OCR piece: resemblance to operations first, scale second.

Link to the LoRA case and sibling posts

The narrow “question classification → label” experiment shows that a small model + clean labels can beat a large checkpoint with a dirty set. Do not port that win one-to-one to open dialogue: the learning unit is not a class but an answer policy, and the cost of “almost right” text is higher.

Reading map:

This piece answers: how to design the corpus once “we will fine-tune” is decided.

Common mistakes

A chat folder with no schema. Mixed systems, mixed formats, no id—zero reproducibility.

Zero refusals. The model learns to be helpful always; in production that is confident hallucination.

90% synthetic with no review. Pretty loss charts, empty pilot.

Training on the eval golden. Metrics cheer until the first unfamiliar query.

One author for all user turns. Gap vs real employee or operator language.

Mixing SFT and preferences in one file. The model cannot tell whether to learn “the only correct answer” or “what people tend to like.”

No manifest. You cannot roll back the corpus or explain a regression.

Ignoring ACL and PII in logs. Legal and security debt in the same JSONL as training.

What to try today

1. Write a one-page pair contract: input, output, refusal, escalation, tone.
2. Draft an intent × condition coverage map and mark empty cells—that is the labeling queue, not “another LoRA run.”
3. Add to the draft corpus at least five refusals of each type from the contract.
4. Run the static gate (schema, dedup, eval overlap) before the first training job.
5. Name an SFT manifest owner—without whose sign-off the file does not enter the job.

FAQ

Next in the series

• Dataset engineering overview — families and lifecycle frame.
• Golden editors — rubric and adjudication (golden-editors-annotation-workflow-2026).
• Train / eval / holdout and leakage (dataset-splits-leakage-2026).
• Preference data: DPO and pairs (preference-dataset-rlhf-2026).
• Synthetic and hybrid pipelines (synthetic-human-dataset-pipeline-2026).