AI Dataset Engineering

Cleaning and normalization

Cleaning and normalization make a consolidated table safe to train on.

Discuss this directionContact form

Tell me the goal, stack constraints, and timeline — I reply on Telegram.

Cleaning and normalization make a consolidated table safe to train on. I fix types, dates, encodings, unicode lookalikes, and the difference between an empty string and a missing value. Cleaning and normalization is not deduplication and not junk removal: a valid rare row stays, a broken encoding does not.

The schema is written in plain language: what each field means, which values are allowed, and what happens to rows that fail. I do not silently coerce a free-text note into a category. Time zones and currencies are named. Text is normalized only where the brief says case, whitespace, or punctuation must not change the label.

Acceptance is a schema, a before-and-after sample, and a count of rows dropped for a named rule. The script fails closed on a new unexpected value instead of guessing. A single table is often 1–2 weeks once collection exists. Duplicates are the next pass — deduplication.

Acceptance criteria

Done when

  • Schema and field meanings are written down
  • Train / validation / test split is reproducible and checked for leakage
  • Quality report lists counts, removed duplicates, and known gaps

Deliverables

  • Dataset files in the agreed format
  • Reproducible preparation script
  • Quality report

Out of scope

  • Training the model and production deployment
  • Legal opinion on personal data and third-party licenses
  • Annotator volume beyond the agreed sample unless it is in the quote

The final acceptance checklist is confirmed in the brief or contract; the list above is a scope alignment guide.

Ballpark estimate

Scope size
Extras

FAQ

Tap a question to expand the answer.

Will you rewrite our source systems?

No. Cleaning lives in the preparation script. The CRM or ERP stays the system of record.

What if a field has three date formats?

I map the ones we can prove and quarantine the rest with a reason, rather than inventing a date.

Discuss this directionContact form

Tell me the goal, stack constraints, and timeline — I reply on Telegram.

View full service page