Task

The goal was to eliminate the manual transfer of data from paper UZT drafts (ultrasonic thickness control) into a digital format. Operators fill in the header, table, and measurement grid I–IV by hand; errors and ambiguous figures are costly when transmitted to ERP.

The project aims for a local pipeline: photo/scanned form → structured result.json with fields and confidence, plus a report for human review (review.html / review.csv), rather than a "magical" end-to-end solution without oversight.

Readiness criteria: reproducible run on a single file and on batch, field statuses (ok / review / error / empty), evaluation on a golden set, groundwork for legacy-payload integration with the internal system (phase 2).

Solution

A hybrid pipeline CV + OCR + vision LLM was built:

1. Python CV pipeline — preprocessing, QR, table and ink detection, PaddleOCR for printed text, EasyOCR for handwritten measurements.
2. TypeScript orchestration — API calls, schema normalization (zod), artifact collection in data/output/{imageId}/.
3. Vision LLM (OpenRouter) — three passes: header, table rows, measurement grid.
4. Confidence engine — merging signals from OCR + LLM + CV, domain validation, and verify-pass by a second model.
5. Golden set and benchmarks — golden editor in the browser, evaluate, model comparison by accuracy/time/cost; a separate path for fine-tuning TrOCR (dataset, notebook, documentation).

The operator sees highlighting of questionable fields and corrects them before export, rather than accepting "raw" OCR as truth.

Architecture and stack

Orchestration: TypeScript, Node (tsx), sharp, zod, dotenv / undici.

Recognition: Python (requirements.txt), PaddleOCR, EasyOCR, optionally TrOCR + Jupyter notebook for fine-tuning.

LLM layer: OpenRouter (keys in .env), multi-pass prompts tailored to the UZT form structure.

Data: data/golden (templates, labels, jsonl), data/erp-cache, output preprocessed.jpg, cv.json, result.json, review.html, review.csv, placeholder legacy-payload.json.

Documentation: pilot, hardware requirements, OCR vs ERP comparison, runbooks for TrOCR and lab-eval loop.

Language distribution in the repository: TypeScript (48%), Python (37%), HTML (15%), Jupyter (1%).

Key features

• Recognition of a single file and batch mode (recognize, recognize:batch, recognize:cv-only).
• Review report with color coding based on thresholds CONFIDENCE_OK / CONFIDENCE_REVIEW.
• Evaluation on golden set and bootstrap dataset for TrOCR.
• Browser-based golden editor (golden:edit) for marking benchmarks.
• Model benchmarking: accuracy, time, API cost.
• Documented path to ERP integration without mixing pilot and production contracts.

Result

The pilot addresses a key scenario: from a photo of the UZT draft, a verifiable structured result and artifacts for the operator are produced. There is a quality pipeline (golden / evaluate / benchmark) and a plan to fine-tune TrOCR for domain-specific handwritten figures. The project is actively in development; ERP export has been deferred to the second phase through legacy-payload.

The repository is private — the architecture and stack are documented in the portfolio without publishing raw forms and client models.

What I would improve

I would refine the verify-pass and confidence thresholds for calibration on a larger golden set, add e2e on CI with fixtures without API secrets, and stabilize the legacy-payload contract with ERP owners before a wide rollout.