Key takeaways

One vision-language model (VLM) on the full page is not enough. Geometry, empty cells, and crops come first; reading comes second.

Local computer vision (CV) and optical character recognition (OCR) still matter even when handwriting accuracy is weak. In the pilot, CV/OCR alone scored about 35% on handwritten measurements — useless for acceptance, essential for knowing where the table is, where ink is, and where confidence attaches.

Multi-pass vision LLMs plus a second-model verify pass catch silent errors better than one confident answer.

Confidence is a product, not a side-channel score. Signal weights, domain rules, and a hard ban on auto-ok for ambiguous tenths ,5 / ,6 / ,7 beat a clever prompt.

Review HTML/CSV is the product boundary before ERP. Operators correct the doubtful cells; they do not retype the whole sheet.

A golden set and model benchmarks belong in week one of a pilot — otherwise you cannot honestly compare “cheap” and “expensive” modes.

Fine-tuning (TrOCR / CRNN) is an optional path, not a substitute for system architecture. Model choices for digits are covered separately in “from CNN to TrOCR”.

The job and the cost of error

Manual transfer from paper into a digital system is not a UX inconvenience; it is a systematic error source. Handwriting varies, paper wrinkles, phone photos sit at an angle, cells contain smudges or overwritten values. The business does not want “pretty text from the form”; it wants fields with a contract: thickness in range, date in format, element type from a dictionary, empty truly empty.

The cost of a miss is closer to industrial safety and accounting integrity than to a UI typo. So the pilot rejected “upload the JPG to a chat model → paste into ERP.” We need a loop where doubtful values are visible and not accepted silently.

The same idea — low recognition confidence must not quietly travel downstream — shows up in corporate document ingest for RAG. The domain differs (UT draft vs policy PDF); the discipline does not.

Why not one VLM on the whole sheet

It feels natural: modern vision models “see” a page, so why crop? In a industrial pilot, the full sheet is a bad default input.

One page mixes a printed header, a QR code, semi-printed row metadata, and a dense handwritten measurement grid. Zones need different attention: the header has few fields but messy executor names; the grid has hundreds of tiny digits where tenths matter (12.3 vs 12.8). A single full-page call easily smooths empty cells, swaps rows, and emits plausible JSON unbound from geometry.

A known form template is an advantage. After deskew you can find the table, crop cells, and solve narrow subproblems. Same order as in the handwritten-digits article: geometry first, model second.

flowchart LR
  photo[PhotoForm] --> cv[LocalCV_OCR]
  cv --> vlm[MultiPass_VLM]
  vlm --> conf[ConfidenceFusion]
  conf --> review[OperatorReview]
  review --> erp[ERP_Phase2]

CV stage: geometry beats guessing

Stage one runs locally: photo normalization, perspective correction, QR, table detection, ink detection, draft OCR. TypeScript orchestrates; the heavy CV path is Python (PaddleOCR for print, EasyOCR as a handwriting hint).

The job is not to “beat handwriting.” It is to build a cell map: coordinates, ink present or not, preliminary text, crops for later stages. Without that map you have nowhere to hang statuses and nothing useful to show in review.

An honest pilot number: CV/OCR alone was about 35% accurate on handwritten measurements. As acceptance, that fails. As architecture, it is a valid intermediate layer. Empty cells and geometry already justify the stage; “pretty” handwriting is solved elsewhere.

It helps to name operating modes explicitly:

Accuracy and dollar figures below come from internal measurements on a golden sheet in the pilot; other forms will differ while the golden set stays small.

Multi-pass vision LLMs and verify

Stage two sends crops, not the whole page. Three passes:

1. header (date, executor, instrument);
2. row metadata (zone, element type, diameters, design/reject thickness);
3. measurement grid by sections.

For every filled measurement cell a second, independent model runs. If primary and verify disagree, the field does not get a green light. In our confidence model, a handwritten measurement cannot become ok without a verify pass.

That changes the LLM’s role: not the single source of truth, but one signal in a loop that must argue with itself.

In practice this has two engineering consequences. First, cost and latency grow with the number of filled cells, so “max quality” cannot be blindly applied to a whole archive without a budget. Second, prompts and response schemas should differ for header, rows, and grid: one universal prompt for the whole page pulls you back toward the errors cropping was meant to avoid.

Confidence as a product

The final field score is a weighted mix of signals (pilot weights; the code renormalizes over present signals):

Operator statuses (default thresholds):

Domain rules matter as much as the model: thickness in a sane range (pilot defaults roughly 0.5–80 mm), expected date format, element type from a dictionary. Implausibly large OCR numbers often come from fused neighbouring digits — better to clear than to accept quietly.

Hard rule for tenths

In Cyrillic handwriting, tenths ,5, ,6, and ,7 are often indistinguishable. In the pilot those values never auto-receive ok — only review, or error if models also disagree. An extra yellow cell beats a silent millimetre error in a wall thickness.

Review — the product boundary before ERP

A run yields more than JSON. review.html mirrors the form with colour coding; review.csv fits Excel; result.json carries value, confidence, status, and signal detail.

Typical artifacts per photo:

The point is simple: the system removes routine typing, not responsibility. Operators inspect yellow and red. Green can move faster — after thresholds are calibrated on goldens, not after a demo vibe check.

ERP export is deliberately separate. While the contract is unstable, artifacts keep a legacy-payload stub. Mixing recognition pilot and accounting production contract breaks both loops quickly.

Keep roles distinct:

• golden editor — lab quality and labeling;
• operator UI — working review and edits before accounting;
• production edits must not auto-become goldens without explicit promotion.

Otherwise the golden set pollutes and accuracy reports stop being comparable. For golden discipline in another domain, see RAG golden evaluation.

Goldens, evaluation, and benchmarks

Without golden labels, “97%” is marketing. At the time of the cited measurements the pilot golden set was small (a few sheets) and still growing. Say that out loud: the numbers below are an orientation on one golden sheet, not a guarantee across the archive.

Mode comparison on a golden sheet (~313 fields) from pilot measurements:

At ~97% on 313 fields, about 8–10 cells still need manual edit. At ~70%, about 90. Model choice is a labour-and-API budget decision, not a hunt for the one true network.

By zone in the same measurement: row metadata was strongest, the measurement grid strong on the best model, the header (especially handwritten executor) weaker. Architecture should reflect that: different passes, different thresholds, different review expectations.

The public stack can be named without customer detail: TypeScript and Python, response schema via zod, image processing, local OCR, cloud vision models through a compatible API. Roughly half the orchestration code is TypeScript, about a third Python. That matters for teams who think “OCR = one Python script”: an industrial form almost always becomes a service with contracts, reports, and quality evaluation.

What is still open

• ERP integration — phase two; stabilize the export contract separately from recognition.
• Confidence threshold calibration on a larger golden set.
• Verify cost — checking every cell with a second model is expensive; need a policy for when verify is mandatory.
• Fine-tune path (TrOCR / CRNN) — groundwork exists; it does not replace the hybrid loop. See handwritten digits.
• Photo quality — preprocessing helps with angle and shadow; hard glare and crumpled paper still hurt.

Common mistakes on similar forms

Feed the whole page to one model and call it a pilot. You get a demo on one pretty photo and a regression on the archive.

Score only overall accuracy. Without splits by header / rows / grid / empties you will not see where the system actually breaks.

Trust model self-report. “I’m 0.92 confident” without second-model agreement and domain rules is a weak signal.

Auto-accept ambiguous tenths. In thickness control that costs more than an extra operator click.

Mix goldens, production edits, and training data. A month later nobody can explain why “accuracy improved.”

Promise ERP next week before JSON and review are stable. Field and status contract first — bus second.

What to try today

1. Write down the form’s field contract and the cost of error per field type (measurement, date, element type, empty).
2. Build a mini-golden (even a few real photos) and score by zone, not one headline number.
3. Split the loop into geometry → reading → confidence → review; do not skip review because “the model is smart.”
4. Add at least one hard domain rule (range, format, ban auto-ok on known handwriting confusions).

FAQ

Closing

Is it worth a team’s attention on similar industrial forms? Yes — if you are ready to build a loop, not hunt for one magic model. The pilot already runs local CV/OCR, multi-pass vision LLMs, confidence fusion, and operator review. ERP phase, golden-set size, and threshold calibration remain open — and it is fine to say so.

The main practice lesson is simple: a yellow cell for review beats a silent millimetre error.