Why ONNX exists

Training and production live in different worlds. Labs love PyTorch: autograd, debugging, ecosystem. Production needs predictable inference on a server, at the edge, or in a tab. Shipping a full PyTorch interpreter to the browser is unrealistic. You need a shared language: a compute-graph description many runtimes understand.

ONNX (Open Neural Network Exchange) is that language: an open neural-network graph spec. A model from PyTorch (and sometimes other stacks) becomes a file (often .onnx plus external weights) that ONNX Runtime, TensorRT-style pipelines, or browser ORT Web can load.

Without ONNX, typical dead ends:

• “We only have a state_dict — how do we give it to the frontend?”
• “Server uses CUDA, browser uses WebGPU — two codebases?”
• “Transformers.js Hub models are already ONNX — where does our CRNN go?”

Format vs Runtime vs training

Three ideas people mix up:

Analogy: ONNX is closer to an intermediate program representation, Runtime to a VM/JIT for specific hardware, PyTorch to the language you wrote the source in.

In Hugging Face’s browser stack, models often ship as ONNX artifacts already. Transformers.js fetches them into ONNX Runtime Web. Your own model follows the same path: export → verify → only then WebGPU.

What the file looks like on disk

In practice “a model in ONNX” is not always one file. Often it is:

• a single .onnx with graph and weights inside;
• or a graph plus external weight files (better for large nets and partial loading).

On Hugging Face browser cards you usually get artifacts already prepared for ONNX Runtime Web. Your own CRNN lives on your CDN or app static hosting, and you own versioning and cache. Keep a short MODEL.md beside it: opset, input shapes, preprocess, alphabet (for OCR), file hash, export date.

A useful ritual before merge: open the graph in Netron, confirm the input name matches the frontend, and that the output is the expected logits — not a surprise intermediate tensor.

Graphs, operators, and opsets

An ONNX graph is a set of nodes. A node is an op (Conv, MatMul, Softmax, LSTM, …) with tensor inputs and outputs. The operator-set version is the opset (e.g. opset 17). PyTorch export picks an opset; the runtime must implement every node on the chosen provider.

Surprises follow:

• Export on a fresh opset succeeds, but ONNX Runtime Web + WebGPU lacks a rare op → crash or silent fallback.
• A custom PyTorch layer with no standard analogue needs a rewrite from primitives or ONNX symbolic functions.
• The same math can lower to different op sequences — numerically close, not bit-identical.

Tools like Netron help you see the graph: inputs, nodes, black boxes. For export debugging that is mandatory, not decoration.

Exporting from PyTorch

Typical path (simplified):

import torch

model.eval()
dummy = torch.randn(1, 1, 32, 128)  # BCHW matching your preprocess

torch.onnx.export(
    model,
    dummy,
    "crnn.onnx",
    input_names=["input"],
    output_names=["logits"],
    opset_version=17,
    dynamo=False,  # confirm API for your PyTorch version
)

Three decisions matter more than cosmetic flags:

1. model.eval() and no dropout/noisy branches — otherwise the graph and numbers drift.
2. A dummy with the same shape semantics as production (channels, crop height, length).
3. Explicit input/output names — so JS preprocess does not guess tensor order.

Newer PyTorch versions evolve the export path (torch.onnx.export, Dynamo-based modes). Pin the docs for your version in the model README: “exported on torch X.Y, opset Z”.

Right after export:

python -c "import onnx; onnx.checker.check_model(onnx.load('crnn.onnx'))"

And run through onnxruntime in Python before any React talk.

Dynamic axes and shapes

Many models take variable batch or sequence length. In ONNX that is dynamic axes (dynamic_axes at export): e.g. free batch and crop width, fixed height for a CRNN.

Typical mistakes:

• Exported only batch=1, W=128 — another crop crashes in the browser.
• Made everything dynamic — some providers optimize worse.
• JS builds NHWC while the graph expects NCHW — “the model is dumb” with correct weights.

Rule: document the shape contract next to the model (like an API). Browser code must match it exactly.

What breaks on CRNN, YOLO, and custom layers

CRNN / sequences. CTC, recurrent blocks, unusual pooling often yield unsupported or awkward ops. CTC postprocess is frequently kept outside the graph (separately in Python and JS) — easier parity, no fragile alphabet logic inside ONNX.

Detectors (YOLO and kin). Heads, box decode, NMS: some teams export backbone+neck only and keep postprocess outside; others try to fold everything into the graph. For the browser the second path is harder on ops and debugging. Detector engineering context: YOLO pillar.

Custom CUDA kernels / nonstandard modules. If the layer is not in the ONNX opset, export will not “almost work” — replace with standard ops or drop that path for the client.

Readiness bar: on a held-out crop set, OCR strings or box IoU match between PyTorch and Python onnxruntime. Until then, WebGPU is early.

Parity: proving the export is honest

Minimum stand:

1. One preprocess (normalize, resize, channel order) shared by tests.
2. PyTorch run → logits / strings.
3. onnxruntime.InferenceSession run → same metrics.
4. Tolerances: logits via allclose with sane atol/rtol; OCR via exact-match rate on a golden set.
5. Pin versions: torch, onnx, onnxruntime, opset, model file hash.

“0.98 in Colab, garbage in the tab” almost always lands here: different resize, RGB vs BGR, /255 vs ImageNet mean, different quantization. WebGPU is innocent until Python ORT already mismatches PyTorch.

Field OCR quality loops: compact CRNN on UZT measurements.

Mini scenario: from .pt to a number in the UI

Build a narrow loop on one crop type (for example a measurement cell):

1. Trained PyTorch model, eval(), fixed preprocess.
2. Export script to ONNX with named I/O and a chosen opset.
3. Comparison script: 50–200 golden crops → strings or logits in PyTorch and onnxruntime.
4. The same preprocess in TypeScript, load via ONNX Runtime Web in a Worker.
5. WASM provider first, then WebGPU with a fallback.
6. In the UI — cold load time, warm inference, and the provider flag.

That yields a measurable vertical slice instead of an abstract “move all models to the browser”. That slice usually convinces stakeholders better than a WebGPU slide.

Quantization and artifact size

Quantization (FP16, INT8, …) shrinks the file and often speeds inference at a quality cost. It can happen before export, at export, or via ORT tools — what matters is which path you chose and what actually ships to the browser.

For clients, size is a product metric: hundreds of megabytes on first visit hurt cold start, covered in the browser AI article. Same rule: re-run the golden set after quantization.

Runtime providers and the browser path

ONNX Runtime picks an execution provider: CPU, CUDA, TensorRT (when configured), and on the web WASM and WebGPU. One .onnx is theoretically portable; in practice supported ops and speed differ.

Product chain:

PyTorch → export ONNX → ORT Python (parity)
                ↓
        ORT Web + WASM (broad reach)
                ↓
        ORT Web + WebGPU (speed)
                ↓
     Transformers.js or direct ORT API

Calling ORT directly in a Worker fits a custom model without an HF pipeline. Transformers.js fits Hub models and standard pipelines. Both sit in the UI story in WebGPU + Transformers.js.

Checklist: shipping .onnx to production

• Pin torch / onnx / onnxruntime / opset versions
• Export script lives in the repo, not “from a laptop once”
• model.eval(), deterministic dummy matching prod shapes
• Dynamic axes are intentional and documented
• onnx.checker and ORT Python load succeed
• Golden set: parity with PyTorch within tolerance
• Preprocess and postprocess documented beside the artifact (language-agnostic)
• For browsers: verify the target provider (WASM / WebGPU)
• File hash and model update channel (CDN / cache busting)
• Clear what is not in the graph (CTC decode, NMS, rules)

Relation to server inference

Even when the end goal is the browser, server ORT remains a useful baseline. On CPU or CUDA it is easier to catch op gaps, compare speed, and decide whether the model belongs on the client at all. Sometimes the honest outcome is: the model stays on the server, and the browser only runs a light classifier or frame-quality check.

A hybrid looks like this: heavy detection or a large net — API; compact crop OCR — local after the server (or a light client detector) returns a box. ONNX is the shared language for both paths: one export, two providers, one parity table.

Common mistakes

Treating ONNX as an accelerator. The format does not speed anything up; hardware and the Runtime provider do.

Exporting in train mode. Dropout and training branches corrupt the graph and metrics.

Ignoring preprocess. The most common reason “the browser lies”.

Folding all postprocess into the graph without need. Harder export and alphabet/threshold portability.

Not pinning opset and versions. Six months later “the same script” yields a different file.

Jumping to WebGPU before Python parity. DevTools debugging costs more than numpy.allclose.

Agree with the frontend team on an update channel: changing opset or quantization is a new artifact, not a silent CDN overwrite. Otherwise some users keep an old graph in cache, others get a new one, and debugging becomes “works on my machine”. Semantic model filenames (crnn-v3-opset17.onnx) and explicit cache-busting in the URL save weeks.

FAQ

Further reading

• WebGPU + Transformers.js — where .onnx lands in the client.
• YOLO engineering — detection and OCR linkage.
• Compact CRNN for measurements — field OCR quality loop.
• Handwritten digits: CNN, CTC, TrOCR — when a small model is enough.
• Three layers of the LLM stack — graph and runtime in the bigger inference picture.

Conclusion

ONNX is how you freeze a model’s compute graph so many engines can run it. ONNX Runtime is the straightest path from that file to server CPU/GPU or WebGPU in a tab. Export from PyTorch is an engineering contract: opset, shapes, preprocess, parity, versions. Close that contract and client AI stops being a lottery; while the graph is dishonest, no device: 'webgpu' will save you.