Key takeaways

OCR is not one task. Classifying a single digit and recognizing a sequence need different models, metrics, and data.

Document structure beats the neural network. If the form template is known, align geometry and cut cells first — do not feed a whole page to a universal OCR model.

The best dataset looks like production. High MNIST accuracy does not transfer to photos with perspective, shadows, and table lines.

Labels from your database can beat manual annotation. When a photo is tied to a system record, ground truth already exists — you only need a cell image → label pipeline.

CNN is a strong baseline. For one digit in a known cell, a small convolutional net is often faster, cheaper, and easier to debug than a Transformer.

CRNN + CTC is the natural step for sequences. Several digits in one cell without per-character boxes is the classic Connectionist Temporal Classification setup.

TrOCR is not always justified. A general model makes sense for complex text and a wider alphabet; for ten digits in a fixed cell, a specialized solution often wins.

Production is a pipeline. Document detection, perspective correction, validation rules, and human-in-the-loop matter as much as architecture choice.

Two different tasks behind one word: OCR

Before comparing CNN and TrOCR, fix what you are recognizing.

Image classification

One cropped cell holds one digit: image [ 7 ] → class 7. This is 10-class classification (0–9). The model outputs a probability distribution over a fixed label set. Answer length is always one character.

This works when the form template guarantees a single digit in a field, or when you crop so only one symbol remains in frame.

Sequence recognition

Another cell holds several digits: [ 7 8 3 ] → string "783". The model must determine characters, their order, and their count. Output length is not fixed — you need a sequence decoder.

That is a fundamentally different task. A classifier trained on isolated digits will not infer the order of three symbols without a separate segmentation stage — and segmenting handwritten digits on a real form is its own headache.

Why MNIST is a poor proxy for real forms

MNIST offers 60,000 training images of handwritten digits, 10 classes, one digit per image, normalized 28×28 size, controlled capture conditions. Excellent for learning computer vision basics. Dangerous for customer promises.

Real photos of filled forms add factors MNIST lacks:

This is domain gap: a model trained on MNIST can score 99% on the benchmark test and fail on phone photos from the field.

The main rule: the best dataset for your task is data that looks like production. If production is field photos of forms, training should use those photos — not normalized 28×28 glyphs.

The first OCR stage is not a neural network

A common mistake is to search for "the best OCR model" and feed it a full photo. On structured forms, a different order wins:

photo → form detection → perspective correction →
template alignment → cell extraction → preprocessing → OCR

Why a known template is a huge advantage

If the form layout is fixed:

• field coordinates are known or computed after alignment;
• table rows and columns are predictable;
• you do not need to recognize the whole page.

Instead of photo → large OCR model → all text, use photo → geometry → cell → small OCR model. Each cell is a separate input with a small alphabet and bounded length. Complexity drops by an order of magnitude.

Same engineering move as in industrial document ingestion: structure and canon first, then meaning extraction. For forms, "canon" is a aligned image with cropped cells.

OCR only where needed

Universal OCR over the full page spends compute on headers, labels, printed text, and empty fields. On a fixed template:

1. Find form corners (contours, markers, QR).
2. Apply homography — correct perspective.
3. Overlay the template mask.
4. Crop only cells with handwritten readings.
5. Run a narrow model per cell.

The most valuable part is ground truth

Without honest labels, any architecture comparison is noise.

Where correct answers come from

In many applied systems, a form photo is already linked to a database record. An operator photographed a meter reading; the correct value lives in readings. Or a dispatcher entered data after a call — and the photo is evidence.

Then annotation does not need an army of labelers. You need a pipeline:

photo → session/form identification → DB query →
correct field value → (cell image, label) pair

Examples:

• cell_00001.png → "78"
• cell_00002.png → "73"
• cell_00003.png → "71"

Why this beats manual labeling

• Scale: thousands of examples accumulate automatically.
• Fewer errors: no one retypes digits into a side file.
• Freshness: the dataset updates with document flow.
• Business alignment: labels match what the system treats as truth.

Caveat: DB ground truth is honest only if the record is correct. If operators systematically mistype, the model learns the same mistakes.

Data leakage — the silent metric killer

When splitting train/validation/test, do not accidentally mix:

• different crops of the same form;
• photos from one person with nearly identical handwriting;
• shot series under the same lighting.

Random shuffling without grouping yields optimistic 98% on test and disappointment in production. Group by document, author, capture session, or date — whatever is truly "new" in production.

Broader AI evaluation practice is in evaluating enterprise AI; for OCR, group splits and sequence-level metrics are critical.

First model — CNN as baseline

A convolutional neural network is the natural starting point for "one digit in a cell."

How a CNN sees a digit

Typical feature hierarchy:

1. Lower layers — edges, lines, corners.
2. Middle — stroke fragments, arcs, intersections.
3. Upper — a glyph resembling a class.

Pipeline: image → convolutions → pooling → features → dense → 10 classes.

Why CNN is a good baseline

On the first experiment, measure not only accuracy but confusion matrix, inference time, and artifact size. Those numbers argue against "let's use TrOCR right away" in meetings.

Minimal experiment

# Pseudocode: 10-class classifier
model = Sequential([
    Conv2D(32, 3, activation='relu'),
    MaxPooling2D(),
    Conv2D(64, 3, activation='relu'),
    MaxPooling2D(),
    Flatten(),
    Dense(128, activation='relu'),
    Dense(10, activation='softmax'),
])
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

For production, add cell size normalization, inversion for dark backgrounds, and confidence logging (softmax probability).

Several digits in one cell

Cell [ 78 ] breaks the "one class per image" setup. Naive approach:

78 → segmentation → 7 | 8 → two CNN runs → "78"

Why segmentation is its own problem

• Digits differ in size within one cell.
• Spacing between symbols varies.
• Symbols touch or overlap.
• Table lines cross strokes.
• Handwriting ignores a monospace grid.

Each factor adds a stage that needs labels and metrics. Better question: can we recognize the whole sequence at once? That is where CTC appears.

CTC: sequence without per-character boxes

Connectionist Temporal Classification learns alignment between an image and a string without bounding boxes per character. A pair whole cell → "783" is enough.

Intuition

At each time step the model outputs a distribution over symbols plus a blank token. The CTC decoder collapses repeats and removes blanks:

raw:  7 → 7 → blank → 8 → 8 → 8 → blank → 3
CTC:  7 → 8 → 3

You do not draw a box around every digit by hand.

CTC limitations

• Sequence is usually modeled left to right (fine for digits in a cell).
• 2D text layout is harder than with attention seq2seq.
• Feature sequence length must be compatible with string length — rarely an issue for short numbers.

CRNN — the classic OCR workhorse

A convolutional recurrent neural network combines convolutions, recurrence, and CTC:

image → CNN → feature map → sequence → BiLSTM → CTC → "783"

Why an RNN between CNN and CTC

Text has order: 7 then 8 then 3. The CNN feature map can be sliced into vertical strips; each strip is a time step. BiLSTM looks both ways and refines which symbol is likely given neighbors.

Pros and cons

For cells with two to five digits, CRNN + CTC remains a strong choice in 2026 — especially when model size and CPU inference matter.

TrOCR — when Transformers enter

TrOCR (Transformer-based OCR) is end-to-end: a Vision Transformer encodes the image; a Transformer decoder generates text character by character.

image → ViT encoder → visual tokens → text decoder → "783"

CRNN vs TrOCR

Fine-tuning pretrained

Typical recipe: take microsoft/trocr-base-handwritten or similar, fine-tune on your digit cells. Pretrained stroke knowledge transfers; top layers adapt to your domain — table lines, camera noise, local handwriting.

Important: more parameters ≠ better on your sample. A Large model may overfit on a thousand cells while Small matches CER with ten times faster inference.

Why TrOCR can be overkill

Main thesis: if the task is image → "7" in a known cell with a ten-digit alphabet, a full OCR Transformer is overkill.

TrOCR becomes interesting when:

• cell length varies (7835, 12, 0);
• the alphabet grows (letters, decimal point, minus sign);
• capture quality varies a lot and pretraining helps;
• you plan one model for digits and short text in neighboring fields.

If instead:

• the template is fixed;
• the alphabet is only 0–9;
• length is bounded (e.g. at most five characters);
• text region is cut by geometry,

a specialized CRNN or even a CNN cascade often wins on latency, cost, and maintenance.

Experiment: CNN vs CRNN vs TrOCR on one dataset

Architecture comparison only makes sense on the same data with an honest split.

Splitting the dataset

Typical ratio: 70% train, 15% validation, 15% test — but within groups:

• by document_id (all cells of one form in one split);
• by author_id / operator;
• by form series or capture date.

Random cell shuffling without grouping inflates metrics.

Metrics

Single digit (CNN):

• Accuracy;
• Per-class precision / recall;
• Confusion matrix.

Sequence (CRNN, TrOCR):

• CER (Character Error Rate);
• Exact Match Accuracy;
• Sequence Accuracy.

Engineering metrics (often decide production):

Record hardware and batch size — otherwise comparison is meaningless.

Error analysis beats a single accuracy number

accuracy = 98.7% sounds good until you open the confusion matrix. Typical confusions on handwritten digits:

• 1 ↔ 7
• 3 ↔ 8
• 5 ↔ 6
• 0 ↔ 6
• 4 ↔ 9

Beyond the matrix

• Handwriting: which operators or counterparties yield worst CER;
• Writing tool: gel pen vs pencil;
• Lighting: glare on laminated forms;
• Digit size: small handwriting in a narrow cell;
• Capture artifacts: motion blur, clipped cell edge.

Build a hard examples folder — 50–200 cells where all models fail. Retraining on hard sets or manual review often helps more than switching Large for Base.

Improvement loop:

normal examples → val errors → hard examples →
augmentation / retrain / rules → repeat

Observability — confidence logging, saving low-score crops, CER dashboards by field and operator — overlaps with AI observability.

Data augmentation: mimic the camera, not fantasy

Augmentation helps when data is scarce if it mimics real capture:

• small rotation and shift;
• scale;
• contrast and brightness;
• Gaussian noise;
• light blur;
• mild perspective warp;
• random crop keeping the digit in frame.

Bad augmentation invents conditions that never appear in production: extreme warps, meaningless color inversion, random lines not where table lines occur.

If table lines are the main error source, remove them in preprocessing (morphology, color mask, line inpainting) rather than hoping the model adapts.

Semi-supervised learning and human-in-the-loop

When a baseline works, close the loop:

confident prediction → pseudo-label → new training samples → retrain

Confidence threshold

If confidence > threshold (e.g. 0.95 for CNN or low CER beam search for CRNN), add to the pseudo-label pool. Below threshold — human queue.

Human-in-the-loop

model → confidence → high: accept / low: human review → DB

This cuts labeling cost and prevents silent degradation: the share of manual checks is a system health metric.

Watch feedback loops: if the model systematically confuses 5 and 6, pseudo-labels amplify the error. Periodically mix in human-verified examples and monitor confusion on fresh traffic.

Production pipeline: more than a neural network

End-to-end chain for a structured form:

photograph → document detection → perspective correction →
template alignment → cell extraction → preprocessing →
OCR model → confidence → validation rules → database

Where deterministic rules win

The network should not solve what business logic already defines:

• field allows digits only;
• value in range 0–100;
• fixed digit count (e.g. meter reading — exactly five digits);
• checksum or cross-field consistency.

If OCR returns 783 for a field capped at 100, a rule rejects it regardless of softmax.

Model cascade

You do not have to pick one architecture:

fast CNN → confident? → accept
           ↓ no
      CRNN or TrOCR → confident? → accept
           ↓ no
      human review

Why this often beats one large model:

• cheaper average cost per cell;
• faster on easy cases;
• easier load scaling;
• hard cases get more compute, not every case.

What to choose for a real system

Architecture map:

ONE DIGIT           → CNN
SEVERAL DIGITS      → CRNN + CTC
COMPLEX TEXT        → TrOCR
STRUCTURED DOCUMENT → Document AI + geometry + per-cell OCR

What this task teaches about model choice

Do not start with "what is the most powerful model today?" Start with "what is the structure of my task?"

• 10 classes, one digit — no generative OCR needed.
• Symbol sequence without per-character boxes — CTC / CRNN.
• Complex text and pretrained transfer — Transformer models like TrOCR.
• Known template — document geometry removes half the ML complexity.

Architecture should follow information structure, not model hype.

What to do today

1. Fix the formulation: one digit or a string? Max length?
2. Draw the pipeline before ML: form alignment and cell cutting first.
3. Check label source: can photos link to the database?
4. Train a CNN baseline and save the confusion matrix.
5. If cells hold several digits — CRNN + CTC on the same crops.
6. Compare TrOCR Small only after a document-grouped split.
7. Add a confidence threshold and a human review queue before full automation.

Further reading

• RAG document ingestion: PDF, tables, OCR
• Production RAG engineering
• Evaluating enterprise AI
• AI observability
• Chartographer and VLM for charts

FAQ