The ONNX Deployment Pipeline

A trained neural network is a tuple of (architecture, weights, operator semantics). The architecture lives in Python source code, the weights live in a framework-specific tensor format, and the operator semantics live implicitly in the framework's C++ kernels. Deploying a model means decoupling all three from the training framework — and that is the problem ONNX was designed to solve.

Throughout the rest of this page, every concept is grounded in one concrete model: a small but realistic CNN trained on CIFAR-10. CIFAR-10 is 60,000 32×32 RGB images across 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck). Small enough to animate end-to-end, complex enough to exhibit every interesting deployment phenomenon.

import torch.nn as nn

class CIFAR10Net(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        # Block 1: 32x32x3 -> 16x16x32
        self.conv1 = nn.Conv2d(3,  32, 3, padding=1)
        self.conv2 = nn.Conv2d(32, 32, 3, padding=1)
        self.bn1   = nn.BatchNorm2d(32)
        self.pool1 = nn.MaxPool2d(2)

        # Block 2: 16x16x32 -> 8x8x64
        self.conv3 = nn.Conv2d(32, 64, 3, padding=1)
        self.conv4 = nn.Conv2d(64, 64, 3, padding=1)
        self.bn2   = nn.BatchNorm2d(64)
        self.pool2 = nn.MaxPool2d(2)

        # Block 3: 8x8x64 -> 4x4x128
        self.conv5 = nn.Conv2d(64, 128, 3, padding=1)
        self.bn3   = nn.BatchNorm2d(128)
        self.pool3 = nn.MaxPool2d(2)

        # Classifier head
        self.fc1 = nn.Linear(128*4*4, 256)
        self.drop = nn.Dropout(0.5)
        self.fc2 = nn.Linear(256, num_classes)

    def forward(self, x):
        x = self.pool1(F.relu(self.bn1(
              self.conv2(F.relu(self.conv1(x))))))
        x = self.pool2(F.relu(self.bn2(
              self.conv4(F.relu(self.conv3(x))))))
        x = self.pool3(F.relu(self.bn3(self.conv5(x))))
        x = x.flatten(1)
        x = self.drop(F.relu(self.fc1(x)))
        return self.fc2(x)

Layer-by-layer tensor shapes

Exporting is not a translation. It is a reconstruction: PyTorch's forward() is arbitrary Python, and ONNX is a static dataflow graph. Bridging the two requires either running the Python (tracing) or parsing it (scripting), and each has failure modes.

dummy = torch.randn(1, 3, 32, 32)

torch.onnx.export(
    model, dummy, "cifar10.onnx",
    input_names=["image"],
    output_names=["logits"],
    dynamic_axes={
        "image":  {0: "batch"},
        "logits": {0: "batch"}
    },
    opset_version=17,
)

scripted = torch.jit.script(model)
torch.onnx.export(scripted, dummy, "cifar10.onnx")

The dynamic axis dance

By default, tracing bakes the dummy input's shape into the graph. A model traced with (1,3,32,32) will only accept batch size 1 unless you mark the batch dimension as dynamic via dynamic_axes. Most production deployments require at least {0: "batch"} on every input and output.

Modern alternative: torch.export

PyTorch 2.x introduced torch.export, a compiler-grade frontend that produces a fully captured FX graph with symbolic shapes. The new ONNX exporter (torch.onnx.dynamo_export) builds on it, eliminating most edge cases of the legacy tracer.

# The 2.x dynamo path
from torch.onnx import dynamo_export

prog = dynamo_export(model, dummy)
prog.save("cifar10.onnx")

An .onnx file is a serialized ModelProto message. Understanding the schema is the difference between treating ONNX as a black box and being able to debug, patch, or hand-author models.

ModelProto {
  ir_version,               // e.g. 9
  producer_name, version,
  opset_import [{domain, version}],
  graph: GraphProto {
    name,
    node:        [NodeProto],   // the ops
    initializer: [TensorProto], // weights
    input:       [ValueInfo],
    output:      [ValueInfo],
    value_info:  [ValueInfo]    // intermediates
  },
  metadata_props
}

NodeProto {
  op_type:    "Conv",
  domain:     "",            // "" = standard
  name:       "/conv1/Conv",
  input:      ["image",
               "conv1.weight",
               "conv1.bias"],
  output:     ["/conv1/Conv_out"],
  attribute:  [
    {name: "kernel_shape", ints: [3, 3]},
    {name: "pads",         ints: [1, 1, 1, 1]},
    {name: "strides",      ints: [1, 1]},
    {name: "group",        i:    1}
  ]
}

Inspecting our exported CIFAR-10 graph

import onnx
m = onnx.load("cifar10.onnx")

print(f"opset: {m.opset_import[0].version}")
print(f"ir_version: {m.ir_version}")
print(f"nodes: {len(m.graph.node)}")
print(f"initializers: {len(m.graph.initializer)}")

for n in m.graph.node[:5]:
    print(n.op_type, n.input, "->", n.output)

# Conv ['image', 'conv1.weight', 'conv1.bias'] -> ['/conv1/Conv_out']
# Relu ['/conv1/Conv_out']                       -> ['/Relu_out']
# Conv ['/Relu_out', 'conv2.weight', ...]        -> ['/conv2/Conv_out']
# BatchNormalization [...]                       -> ['/bn1/BN_out']
# Relu ['/bn1/BN_out']                           -> ['/Relu_1_out']

Visualizing the graph

Every NodeProto is a vertex; every shared tensor name is an edge. Here is the topological structure of our CIFAR-10 graph after export, before any optimization:

Initializers vs inputs

A subtlety that trips up newcomers: weights live in graph.initializer, not graph.input. Some tools list them under both for backward compatibility, but the canonical interpretation is: anything in initializer is a constant tensor baked into the model; anything in input is something the caller must supply at runtime.

An opset version is the contract between producer and consumer. Conv-11 and Conv-22 may have different attribute defaults, broadcasting rules, or supported dtypes. Pinning an opset is as load-bearing as pinning a Python version.

Why opset version matters: a Resize example

Consider a model with image upscaling. In opset 10, Resize took scales as a single input. In opset 11, the signature changed: roi was added, and scales moved to the third input. A model exported under opset 10 will silently mis-link arguments if loaded under a strict opset-11 importer.

Custom operators

When the standard opset doesn't cover an op (e.g., a fused attention with bespoke masking), you have three choices:

An exported graph is rarely the graph that actually runs. Between .onnx on disk and the first kernel call sits a graph optimizer that can shrink the node count by 30–60% and the latency by 2–5×.

Three classes of transformation

# Before
y = Reshape(weight, Concat([Shape(weight)[0:2], Constant([3,3])]))

# After
y = precomputed_constant

W' = W * (γ / sqrt(σ² + ε)).reshape(-1, 1, 1, 1)
b' = (b - μ) * γ / sqrt(σ² + ε) + β

Quantization replaces FP32 weights and activations with lower-precision types — typically INT8 or FP16. The weight file shrinks 4×, integer SIMD throughput goes up 2–4×, and on accelerators with INT8 tensor cores the speedup compounds. The price is accuracy loss, which good quantization keeps under 1% top-1.

The math: affine quantization

An INT8 tensor q approximates an FP32 tensor x via a per-tensor (or per-channel) scale and zero point:

x ≈ scale · (q − zero_point)
where  q ∈ [−128, 127]  (int8, signed symmetric)
   or  q ∈ [   0, 255]  (uint8, asymmetric)

scale       = (x_max − x_min) / (q_max − q_min)
zero_point  = round(q_min − x_min / scale)

Per-channel quantization (one scale per output channel of a Conv) preserves accuracy far better than per-tensor on CNNs. ONNX's QuantizeLinear and DequantizeLinear ops both accept either scalar or 1-D y_scale / y_zero_point to indicate this.

QDQ vs QOperator format

Quantized ONNX models come in two flavors:

x → DQ → Conv → Q → DQ → Relu → Q → ...
              ↑
          float weights are
          dequantized JIT

x_int8 → QLinearConv → y_int8
        (W_int8, scales, zero_points
         baked into op inputs)

Quantizing CIFAR-10Net (live walkthrough)

from onnxruntime.quantization import quantize_static, CalibrationDataReader

class CIFAR10Calib(CalibrationDataReader):
    def __init__(self, samples=512):
        self.it = iter([{"image": x.numpy()}
                          for x, _ in dl][:samples])
    def get_next(self): return next(self.it, None)

quantize_static(
    "cifar10.onnx",
    "cifar10_int8.onnx",
    CIFAR10Calib(),
    quant_format=QuantFormat.QDQ,
    activation_type=QuantType.QInt8,
    weight_type=QuantType.QInt8,
    per_channel=True,
)

ONNX Runtime (ORT) is the reference implementation. Its architecture is worth studying because nearly every other runtime — TensorRT, OpenVINO, even mobile-only ones — follows the same broad shape.

The session lifecycle

1. Load. Parse .onnx protobuf into in-memory Graph.
2. Optimize. Apply graph transformers (Level 1: trivial, Level 2: fusion, Level 3: layout). User picks the level.
3. Partition. Walk the graph. For each node, ask each enabled EP "can you take this?" The first EP that can claim a connected subgraph gets it.
4. Compile. Each EP compiles its assigned subgraph into a kernel sequence (or a single fused kernel, in TensorRT's case).
5. Plan memory. Compute lifetimes for every intermediate tensor; allocate a single arena that all intermediates alias into.
6. Run. For each call to session.run(), walk the partitioned plan and dispatch.

import onnxruntime as ort

opts = ort.SessionOptions()
opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
opts.intra_op_num_threads = 4
opts.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL

sess = ort.InferenceSession(
    "cifar10_int8.onnx",
    sess_options=opts,
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)

# IO binding avoids host↔device copies on hot paths
io = sess.io_binding()
io.bind_input("image",  "cuda", 0, np.float32, [1,3,32,32], img.data_ptr())
io.bind_output("logits", "cuda")
sess.run_with_iobinding(io)

An execution provider is the bridge between the runtime's IR and a hardware-specific kernel library. Each EP is a plug-in that registers (1) which ops it implements, (2) capability metadata, and (3) compiled kernels.

Partitioning in practice

Suppose we run our CIFAR-10 model with ["TensorRTExecutionProvider", "CUDAExecutionProvider", "CPUExecutionProvider"]. The partitioner walks the graph:

TensorRT EP specifically

TensorRT is unusual: it doesn't implement individual op kernels at runtime. Instead it accepts an entire subgraph, builds a fused engine via its own optimizer (kernel auto-tuning, INT8 calibration, fusion across ~50 ops), and serializes it. ORT caches that engine on disk and on next session load skips rebuilding.

providers = [("TensorrtExecutionProvider", {
    "trt_engine_cache_enable": True,
    "trt_engine_cache_path": "./trt_cache",
    "trt_int8_enable": True,
    "trt_int8_calibration_table_name": "cifar10_calib.cache",
    "trt_max_workspace_size": 2<30,
})]

"Deploying a model" means very different things on a 96-vCPU Linux server, an iPhone, a Raspberry Pi, and a browser tab. The same .onnx file should — and with care, can — serve all of them.

Cross-target consistency

One trap: the FP32 reference, the INT8 server engine, and the INT8 mobile engine can produce slightly different logits on the same input. ULP-level numerical differences (different cuDNN algorithm, different rounding mode, different SIMD reduction order) compound through layers. For most applications this is invisible; for safety-critical ones it must be characterized.

A single number ("3 ms") tells you almost nothing. Inference performance is a distribution, parameterized by batch size, thread count, sequence length (for transformers), warmup state, and contention.

The four numbers that matter

Doing it right

1. Warm up. Throw away the first ~50 calls. Lazy compilation, allocator priming, kernel autotuning all happen in the first few runs.
2. Pin clocks. On a GPU, lock to base clock with nvidia-smi -lgc; on CPU, disable turbo or pin frequency. Otherwise variance dominates signal.
3. Isolate the process. No other tenants on the device. Use taskset / numactl on CPU.
4. Measure end-to-end. Include host→device copy if your real workload pays it. io_binding with pre-resident GPU tensors can hide a real cost.
5. Vary load. Latency at QPS=1 and at QPS=saturation are different curves. Most production systems live in the knee of that curve.

import time, numpy as np, onnxruntime as ort

sess = ort.InferenceSession("cifar10_int8.onnx",
                            providers=["CPUExecutionProvider"])
x = np.random.randn(1,3,32,32).astype(np.float32)

# warmup
for _ in range(50): sess.run(None, {"image": x})

t = []
for _ in range(1000):
    s = time.perf_counter_ns()
    sess.run(None, {"image": x})
    t.append(time.perf_counter_ns() - s)

t = np.array(t) / 1e6  # ms
print(f"p50 {np.median(t):.3f}  p95 {np.percentile(t,95):.3f}  p99 {np.percentile(t,99):.3f}")

A deployed model is an attack surface. The model file itself can carry executable code via custom ops. The runtime can be made to leak inputs through timing. The training pipeline can be poisoned upstream of export. ONNX deployment teams should treat the .onnx file with the same scrutiny as any third-party binary.

Threat model overview

A worked example: timing side channel on CIFAR-10

Suppose CIFAR-10Net is served behind an HTTP API that returns predicted class. An attacker sends 10,000 inputs and records the precise latency of each response. Even with all per-class compute paths fused into a single graph, attacker-observable latency variance correlates with predicted class because:

• The post-softmax argmax branch on the host returns earlier when one logit clearly dominates.
• Cache effects: classes whose decision boundaries hit hot regions of the FC weight matrix have lower L2 misses.
• The HTTP serializer's response length depends on label string length ("airplane" vs "cat").

None of this is exploitable on a per-call basis. With 10,000 calls and statistical analysis, label distributions become recoverable. The fix is application-layer: respond at a fixed deadline (e.g., always return at T+5ms), not when computation finishes.

Closing the loop: a hardening checklist

1. Sign and verify every .onnx at load time. Reject unsigned models in production.
2. Disable custom op libraries; whitelist domains your team owns.
3. Build the runtime with the minimal set of EPs needed; smaller binary = smaller surface.
4. Run the runtime under a sandboxed user, with seccomp / AppArmor restricting syscalls to the strictly required.
5. Pad responses to a fixed deadline; truncate output to the minimum information clients need.
6. Maintain a golden test set; alert on drift between deployed variants and FP32 reference.
7. Treat the training-to-export pipeline as production code: review, CI, reproducible builds, signed artifacts.

If you trace one mental model through this entire pipeline — a 32×32 image entering a Python forward(), becoming a frozen graph of ~200 operator definitions, undergoing fold-fuse-quantize until a 580K-parameter network fits in 600 KB, then dispatching across nine possible execution providers down to vendor SIMD intrinsics — you can see why ONNX is more than a file format. It's the impedance match between research-grade flexibility and production-grade efficiency.

Three principles to internalize: