Key takeaways

Cause first, architecture second. Low exact match is almost never cured by “one more layer” until you know: too little data, dirty labels, underfit, overfit, or a wrong problem (empties inside the same head).

You cannot improve what you measure wrongly. For a thickness cell, exact string match and empty false positives matter more than loss and character score. The instrument catalogue lives in the metrics sibling.

One lever per run. Otherwise the experiment log becomes folklore.

The dataset beats a hyperparameter. Handwriting variety, crop quality, and hard negatives move the number more often than a new optimizer.

Epochs are an honest first lever for underfit. The net may simply not have finished. More epochs ≠ always better: watch the train/val gap.

When a cheap lever stalls, find the bottleneck. After mixing empties, that is no longer “twenty more epochs,” but a tiny empty-versus-value classifier and a human on the tail.

System quality ≠ model quality. A loop with confidence and operator review often beats an endless chase for 99% automation.

Why “train it” is not the end

Training is hit by everything at once: data, architecture, loss, optimizer, learning rate, batch size, epochs, preprocessing, augmentation, regularization, validation scheme. The temptation is to treat that as a mixing desk. The result is a pretty loss curve and a confident 8,0 on an empty cell.

The real question is not “how do I start training,” but how do I systematically raise quality. Same through-line: cell crop → convolution features → bidirectional memory → CTC → string. First compact run: about a thousand numeric values, around 19 minutes, 70.9% exact match and 89% per character. That is the start of the loop, not shop-floor acceptance.

flowchart TB
  data[Dataset_and_labels]
  data --> train[Train]
  train --> measure[Measure_exact_and_empty_FP]
  measure --> why[Error_analysis]
  why --> hyp[One_hypothesis]
  hyp --> change[Change_one_lever]
  change --> train

What “quality” means here

CTC loss going down means the net aligns sequences better. That is not the same as a correct thickness. Gold 58321, prediction 58327: loss may have fallen, the measurement is wholly wrong. The operator and ERP need the string.

For OCR in a measurement cell we watch:

• exact string match;
• character accuracy (diagnosis, not acceptance);
• false positives on empties;
• separately — narrow columns and hard crops.

We do not retell Accuracy / F1 / CER / WER: that map is in quality metrics. The trap in our validation script matters more: train_crnn.py skips empty gold labels. Exact match is counted only on filled cells. You can “improve” 70.9% and grow ghosts on empties unless a second report exists. That is why the first note kept empty false positives in their own column.

Diagnose first, not a new net

Low quality branches.

too little / too similar data
dirty or disputed labels
underfitting
overfitting
learning rate
crop preprocessing
wrong problem statement

Do not change the architecture first. In the pilot the head is already narrow: input 48 × 160, about 640 thousand parameters, AdamW 1e-3, weight decay 1e-4, batch 16, grad clip 5, ReduceLROnPlateau, early stop with patience 18 at a cap of 80 epochs. That is enough contour to read the logs first: train and val loss, exact match, whether one rises while the other falls.

Typical pictures.

Error analysis after every run beats a new optimizer. Look at which cells break: last digit, comma, fat grid, faint pencil, a ditto that became 1. That is a hard-example queue, not a reason to bolt on attention and a Transformer tonight.

The dataset is the main lever

A thousand similar crops from one hand beat several hundred sheets from many hands — in the wrong direction. Volume without variety fakes learning. Dataset-as-product lives in a separate series; here — what already cuts measurement-head quality.

Labels. A pair “image → 12,6” when the true value is 12,5 teaches confident lying. A small share of such pairs hits CTC harder than the “wrong” batch size. In the pilot disputed crops get a quality tag (clean / hard); training may include hard ones, a clean eval should not. That is discipline, not a hyperparameter.

Balance. A rare “last-digit” failure is not cured by a mean over all symbols. If nines are scarce, the net reads eights and fumbles the tail. The error queue after a run is honester than blindly oversampling a class.

Hard examples. The loader already has include_hard. The point is not “add noise,” it is to put back the crops the current checkpoint fails. Cycle: train → error list → label / filter → train again. Without a log this becomes endless fine-tuning on everything.

Mixing empties into the same head is a separate lever we already measured: ghosts fall, digits too. That is not “more data,” it is a change of problem. We will not reprint the 85.4% / 100% false-positive table — it is in the first note. For the loop: if the bottleneck is empties, the next experiment is not “more epochs on the same mix.”

Preprocessing and augmentation — only as real input

The same head on a different crop is a different quality. In the pilot the chain is narrow: grey, autocontrast, canvas 48 × 160, invert, mild rotate and shift, narrow contrast. That mimics a phone and a grid, not “creative noise.”

Rule: augmentation must look like what arrives from the form. Tilt, shadow, slight blur — yes. Strong elastic warp the operator never produces — a way to learn a world that does not exist and tank validation.

Crop-quality policy (legacy / strict) is preprocessing in a wide sense: a bad cut must not ride into training as “one more 12,3.” If cell geometry dances, fix vision first, not the number of filters.

Epochs: the net may simply not have finished

The first compact run sat near 70.9% exact match. In the next cycle we kept the same contour and let the head walk further in epochs: exact match rose to 75.8%. That is not scheduler magic and not a new architecture. Weights were not yet in the well; gradient descent had not finished.

Default in code: cap 80 epochs, early stop if val exact match does not grow for 18 epochs. The scheduler halves the rate if val loss does not drop for 5 steps, floor 1e-5. Worth checking whether we stopped early on patience, or hit a data ceiling.

More epochs are not always better.

train   ████████████  high
val     ████░░░░░░░░  lagging

If the gap grows, you are memorising particular sheets, not learning handwriting. Then the lever is not “40 more epochs,” but augmentation, decay, variety, and early stop on the best snapshot, not the last.

Rate, batch, optimizer — after the cause

Learning rate is step length on the loss surface. Too large — loss jumps, the net overshoots a short-string optimum. Too small — 80 epochs are not enough, and you decide “the architecture is weak.” In the pilot we start at 1e-3 and decay on plateau. Changing it in the same run as batch and epochs is a forbidden journal move.

Batch 16 is a compromise of gradient stability and sample size, not a religion. A smaller batch does not “generalize better by itself”; the effect couples to rate and noise. Until data and epochs are closed, comparing 8 / 32 / 64 is an expensive queue.

AdamW with weight decay is already there. Switching to plain SGD “because a 2014 paper” without a hypothesis is noise. Regularization already sits in three places: decay, memory dropout 0.2, early stop. Adding another without seeing overfit is pointless.

Transfer from a large convolution and an ensemble of checkpoints are not done in the pilot. While the bottleneck is empties and handwriting variety, those are late levers. We write that honestly: we did not run them, we do not invent plus three points.

The log: one row, one change

Without a log the cycle falls apart. A minimal pilot table (not a lab bench of every optimizer):

The empty rows are already a problem-statement conflict, not a training victory. They stay in the log so we do not repeat “fine-tune on empties until victory.” We do not backfill augmentation or scheduler rows: there was no isolated “scheduler only” run with an honest delta.

A draft plan makes it easy to draw a pretty staircase “epochs → augmentation → scheduler → 80%.” Without an isolated run that is decoration. We do not publish those steps: there is nothing to confirm them. A shorter log is more honest than an encyclopedia.

Rule: do not turn five knobs at once. If 70.9% → 75.8% with everything else equal, you may attach the gain to epochs. Change data, rate, and depth together — you get a story you cannot replay.

When quality stalls — and when to stop squeezing the model

After 70% gains are relatively cheap. After 80% errors become rare and pathological: fat grid, a new hand, a bad crop. The last points often need a different system, not a new head: empty valve, 80 mm shield, review status, a human.

In the pilot the cheap epoch lever no longer closes ghosts. The next bottleneck is cell kind before OCR and a tiny empty-versus-value classifier, not an ensemble of three CRNNs. Until that classifier exists, keep the tail in review. That is already the form-level note: the model does not have to answer every time.

Sometimes “95% automatic + doubtful to a human” beats a production system that silently chases 99%. The cost of a missed thickness is higher than an extra glance — same frame as testing economics.

Typical mistakes

The most common is twisting learning rate, batch and augmentation in one run, then arguing “what worked.” A one-row-per-change log is cheaper than a second checkpoint.

The second is watching only the loss. CTC can fall while the last digit keeps lying. Exact match and empty ghosts belong next to it, not “later in the exec report.”

The third is fine-tuning the same head on empties because ghosts feel insulting. We already measured: digits fall too. If the bottleneck is an empty field, the next experiment is cell kind, not twenty more epochs.

The fourth is comparing architectures on different sets. A new hand in train and the old val look like “the Transformer is better,” when you simply added data.

The fifth is chasing automatic 99% while the operator still stares at a grey highlight. A confidence threshold closes the shop floor sooner than a perfect head.

What to try today

Start a five-column log: hypothesis, what you did not touch, exact match, empty ghosts, decision (keep / revert).

Run the current checkpoint on errors and write down ten cells that fail the same way. That is the next dataset, not “more random crops.”

If train and val are both low — add epochs or check whether you stopped early. If val lags — do not add layers.

Do not compare architectures until labels, epochs, and one hyperparameter are closed on the same set.

FAQ

Further reading

How the head is built and why empties break CTC — compact CRNN. What to do with a cell before training — cell-content classifier. Metrics outside our grid — neural-net quality. Train/eval hygiene — dataset engineering. Human in the loop on the whole sheet — one model is not enough.

Conclusion

Quality of a compact CRNN on a UT form does not grow from hitting train again. It grows from a loop: measure → see the error → change one lever → measure again. In the pilot an honest 70.9% exact-match baseline moved to 75.8% via epochs; the empty-cell conflict showed that the next lever is no longer inside the same head.

A practical step this week is not a new architecture. It is a one-change log and a list of ten repeating mistakes. If the list is empty, you have not looked at predictions — you have looked at an average.