Benchmarking

Quantization is a trade-off. Before shipping, measure on your model and your deployment hardware:

Latency: is inference actually faster?
Size: did the file shrink?

benchmark() runs locally, on your own machine, not on the platform. That is deliberate: latency numbers only mean something on the hardware you deploy to. Local benchmarks are free and do not consume plan quota.

benchmark() measures speed and size only. Even on real inputs it never checks whether the quantized model is still correct. See Accuracy is a separate job.

Built-in benchmark

Point the benchmark at two ONNX files and the data you want them timed on. It cycles through your eval_data across the timed runs, times both models on whichever ONNX Runtime provider is available on your machine, and reports the hardware it ran on.

from r3alai.quant import Quantizer

# Real inputs, preprocessed exactly as your deployment pipeline preprocesses them
eval_data = [preprocess(p) for p in Path("val_images").glob("*.jpg")]

result = Quantizer().benchmark(
    original_model="model.onnx",
    quantized_model="out/model.quantized.onnx",   # the ONNX file, not the bundle dir
    eval_data=eval_data,
    warmup_runs=5,
    benchmark_runs=20,
)
print(result.latency_ms_original, result.latency_ms_quantized, result.latency_speedup)
print(result.metadata["hardware"], result.metadata["input_shape"])

quantized_model is the path to a .onnx file. If you unpacked a downloaded bundle, that is <bundle>/model.quantized.onnx — see the manifest for what else the bundle contains.

Notes on the input

Pass an iterable of samples: NumPy arrays, PyTorch tensors, or anything np.asarray can turn into a float32 array.
Every sample must match the shape the model expects. Batch size comes from your data: to time batch 32, pass samples of shape (32, 3, 640, 640).
A 3-D sample (C, H, W) gets a leading batch axis added, so (3, 224, 224) is fed as (1, 3, 224, 224).

Argument	Default	Purpose
`eval_data`	required (unless `use_synthetic=True`)	Inputs to time both models on
`warmup_runs`	`5`	Untimed runs before measurement, per model
`benchmark_runs`	`20`	Timed runs, per model

You can reuse the same preprocessing function you used for calibrating the model. If no data samples are available and the network makes the input go through a fixed number of layers, latency benchmarking is easier with synthetic data - more on this in the next section.

Synthetic input

Without data at hand, set use_synthetic=True and omit eval_data. The benchmark then reads the input shape from the model itself and synthesizes random inputs.

If any axis is symbolic (batch, sequence, …), that quantity is set to 1. Use input_shape when the inferred shape is not the one you deploy at.

result = Quantizer().benchmark(
    original_model="model.onnx",
    quantized_model="out/model.quantized.onnx",
    use_synthetic=True,
    input_shape=(32, 3, 640, 640),    # image batch size set to 32
)

Argument	Default	Purpose
`use_synthetic`	`False`	Generate input instead of taking `eval_data`
`input_shape`	inferred from the model	Override when the graph has dynamic axes
`num_samples`	`5`	How many distinct input tensors of shape `input_shape` to generate and cycle through

Terminal output

With verbose=True (the default) the benchmark prints each run’s latency and tracks the running average:

[benchmark] plan: 2 models x (5 warmup + 20 timed runs) over 12 provided sample(s) of shape (1, 3, 224, 224) on arm. Press Ctrl+C to stop early.
[benchmark] quantized: warming up (5 runs)...
[benchmark] quantized: run 12/20 · sample 12/12 · last 4.31 ms · avg 4.52 ms
...
[benchmark] summary
  original:  9.14 ms/inference (20 runs, CPUExecutionProvider)
  quantized: 4.48 ms/inference (20 runs, CPUExecutionProvider)
  speedup:   2.04x
  hardware:  arm (Darwin arm64)
  input:     (1, 3, 224, 224) x 12 provided samples

The output is a `BenchmarkResult`

Field	Meaning
`latency_ms_original` / `latency_ms_quantized`	Mean per-inference latency in ms
`latency_speedup`	original / quantized
`metadata["hardware"]`	CPU, GPU (when present), OS, architecture
`metadata["provider_quantized"]` / `_original`	ONNX Runtime execution provider actually used
`metadata["input_shape"]`	Shape the models were actually fed
`metadata["input_source"]`	`"eval_data"` or `"synthetic"`
`metadata["num_samples"]`	Number of samples cycled through (`len(eval_dat)`, or `num_samples` when synthetic)
`metadata["runs_completed_*"]`	Timed runs completed per model
`metadata["early_stopped"]`	Whether the run was interrupted

Benchmark on hardware that matches your deployment target.

Accuracy is a separate job

The benchmark method does not report accuracy. To sign off on a quantized model, please run your task metric on a held-out labeled set, with the same preprocessing step.

Get Started

Core Concepts

Guides

Built-in benchmark

Notes on the input

Synthetic input

Terminal output

The output is a `BenchmarkResult`

Accuracy is a separate job

Next steps

Calibration

Quantization methods

​Built-in benchmark

​Notes on the input

​Synthetic input

​Terminal output

​The output is a BenchmarkResult

​Accuracy is a separate job

​Next steps

Calibration

Quantization methods

Built-in benchmark

Notes on the input

Synthetic input

Terminal output

The output is a `BenchmarkResult`

Accuracy is a separate job

Next steps