AI Dataset Engineering

Junk removal

Junk removal takes out rows that should never teach a model: empty files, repeated headers, OCR garbage, spam,…

Discuss this directionContact form

Tell me the goal, stack constraints, and timeline — I reply on Telegram.

Junk removal takes out rows that should never teach a model: empty files, repeated headers, OCR garbage, spam, placeholder “test” records, and boilerplate that appears on every page. Junk removal is not deduplication. A duplicate of a good example is still a good example; junk has no label worth learning.

Rules are named before they run. “Too short”, “mostly symbols”, “known test account”, “template footer”. I review a sample of what the rule would drop, including a few rows it should keep, so a rare class is not swept out with the trash. Personal data that must leave the corpus is masked or dropped under the brief, not mixed into a vague “clean it up”.

Acceptance is the rule list, the drop counts, and a signed sample of removed and kept rows. The side file of removed keys stays with the dataset. A rules pass on one text source is often 1–2 weeks. Quality numbers come later in dataset quality control.

Acceptance criteria

Done when

  • Schema and field meanings are written down
  • Train / validation / test split is reproducible and checked for leakage
  • Quality report lists counts, removed duplicates, and known gaps

Deliverables

  • Dataset files in the agreed format
  • Reproducible preparation script
  • Quality report

Out of scope

  • Training the model and production deployment
  • Legal opinion on personal data and third-party licenses
  • Annotator volume beyond the agreed sample unless it is in the quote

The final acceptance checklist is confirmed in the brief or contract; the list above is a scope alignment guide.

Ballpark estimate

Scope size
Extras

FAQ

Tap a question to expand the answer.

Will aggressive filters shrink a small class?

They can. That is why a human looks at the sample before the rule is frozen.

Is boilerplate always junk?

Not if the brief says the header is the signal. The rule follows the task, not a generic cleaner.

Discuss this directionContact form

Tell me the goal, stack constraints, and timeline — I reply on Telegram.

View full service page