What happened

Clean benchmarks help prototypes; production inputs are ambiguous, labels noisy, context truncated, and rare edges become frequent failures. For secret scanning the question was: can we cut developer noise while keeping enough recall that real credentials are not suppressed?

Criteria split three ways: primary outcome (precision / fewer false positives), safety constraint (recall inside a band), operational guardrails (latency, cost, reliability, compatibility). Offline runs worked like integration tests—version prompt, model, dataset, pipeline; change one major variable at a time. Eval inputs had to look like prod: a candidate beside distracting lookalikes, not a sterile single string.

Production labels were signals, not ground truth—“dismissed alert” ≠ false positive. Synthetic and open sets filled coverage gaps. Failures were tagged by source (model, prompt, input, pipeline, dataset, label). An LLM-as-judge triaged review load but did not replace humans on ambiguous cases. Offline, they report ~95% false-positive reduction while holding recall within the guardrail.

Why it matters

Many teams twist prompts until a benchmark number moves—then ship a regression. The post frames evaluation as product + data + reproducibility + error analysis. Without that, a “better model” may simply solve a different task.

In practice

1. Write the decision in one sentence and what must not regress (for security, recall is a stop-gate).
2. Version prompt, model, dataset, and config; compare to a known baseline.
3. Change one major variable per run—or you will mis-attribute wins.
4. Keep offline inputs close to production: noise, truncation, nearby distractors.
5. Bucket false positives and negatives—aggregates hide the next fix.
6. Use LLM-as-judge to prioritize review queues, not as ground truth.

Takeaway

Offline eval does not prove every production path, but it supplies structured evidence for a careful online experiment. Without an explicit product frame, model tuning is a lottery.