AI Dataset Engineering

JSONL, Parquet, and WebDataset

JSONL, Parquet, and WebDataset are how a prepared corpus leaves my hands.

Discuss this directionContact form

Tell me the goal, stack constraints, and timeline — I reply on Telegram.

JSONL, Parquet, and WebDataset are how a prepared corpus leaves my hands. I pick the format your trainer already reads, not the one that looks modern. JSONL fits row-wise text and chat records. Parquet fits columnar tables and filters. WebDataset fits images packed with labels into shards a loader can stream.

Each shard has a stable name, a schema or a sidecar that names the fields, and a small reader script that opens the first batch. I do not hide a one-off notebook as the only way in. Large corpora are sharded so a laptop can inspect a slice without downloading everything. The longer note on shards is in the blog: JSONL, shards, and the path to the GPU.

Acceptance is one file set that opens, a schema that matches the sample, and a documented command that reads the first shard. Format conversion of an already clean corpus is often 1–2 weeks. The records themselves still come from text datasets or image annotation.

Acceptance criteria

Done when

  • Schema and field meanings are written down
  • Train / validation / test split is reproducible and checked for leakage
  • Quality report lists counts, removed duplicates, and known gaps

Deliverables

  • Dataset files in the agreed format
  • Reproducible preparation script
  • Quality report

Out of scope

  • Training the model and production deployment
  • Legal opinion on personal data and third-party licenses
  • Annotator volume beyond the agreed sample unless it is in the quote

The final acceptance checklist is confirmed in the brief or contract; the list above is a scope alignment guide.

Ballpark estimate

Scope size
Extras

FAQ

Tap a question to expand the answer.

Which format should we choose?

The one your training stack opens today. I will not convert three ways “just in case” inside the same estimate.

Are shards required?

For a small pilot, one file is enough. Shards start when the corpus no longer fits a comfortable copy.

Discuss this directionContact form

Tell me the goal, stack constraints, and timeline — I reply on Telegram.

View full service page