> ## Documentation Index
> Fetch the complete documentation index at: https://docs.r3al.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Benchmarking
> Measure latency and size of the quantized model locally, on your own hardware.
Quantization is a trade-off. Before shipping, measure on **your** model and **your** deployment hardware:
1. **Latency**: is inference actually faster?
2. **Size**: did the file shrink?
`benchmark()` runs **locally**, on your own machine, not on the platform. That is deliberate: latency numbers only mean something on the hardware you deploy to. Local benchmarks are free and do not consume plan quota.
`benchmark()` measures **speed and size only**. Even on real inputs it never checks whether the quantized model is still correct. See [Accuracy is a separate job](#accuracy-is-a-separate-job).
## Built-in benchmark
Point the benchmark at two ONNX files and the data you want them timed on. It cycles through your `eval_data` across the timed runs, times both models on whichever ONNX Runtime provider is available on your machine, and reports the hardware it ran on.
```python theme={null}
from r3alai.quant import Quantizer
# Real inputs, preprocessed exactly as your deployment pipeline preprocesses them
eval_data = [preprocess(p) for p in Path("val_images").glob("*.jpg")]
result = Quantizer().benchmark(
original_model="model.onnx",
quantized_model="out/model.quantized.onnx", # the ONNX file, not the bundle dir
eval_data=eval_data,
warmup_runs=5,
benchmark_runs=20,
)
print(result.latency_ms_original, result.latency_ms_quantized, result.latency_speedup)
print(result.metadata["hardware"], result.metadata["input_shape"])
```
`quantized_model` is the path to a **`.onnx` file**. If you unpacked a downloaded bundle, that is `/model.quantized.onnx` — see [the manifest](/concepts/manifest) for what else the bundle contains.
## Notes on the input
* Pass an iterable of samples: NumPy arrays, PyTorch tensors, or anything `np.asarray` can turn into a float32 array.
* Every sample must match the shape the model expects. **Batch size comes from your data**: to time batch 32, pass samples of shape `(32, 3, 640, 640)`.
* A 3-D sample (`C, H, W`) gets a leading batch axis added, so `(3, 224, 224)` is fed as `(1, 3, 224, 224)`.
| Argument | Default | Purpose |
| ---------------- | ---------------------------------------- | ------------------------------------------ |
| `eval_data` | *required* (unless `use_synthetic=True`) | Inputs to time both models on |
| `warmup_runs` | `5` | Untimed runs before measurement, per model |
| `benchmark_runs` | `20` | Timed runs, per model |
You can reuse the same preprocessing function you used for calibrating the model. If no data samples are available and the network
makes the input go through a fixed number of layers, latency benchmarking is easier with synthetic data - more on this in the next
section.
## Synthetic input
Without data at hand, set `use_synthetic=True` and omit `eval_data`. The benchmark then reads the input shape from the model itself and synthesizes random inputs.
If any axis is symbolic (`batch`, `sequence`, ...), that quantity is set to `1`. Use `input_shape` when the inferred shape is not the one you deploy at.
```python theme={null}
result = Quantizer().benchmark(
original_model="model.onnx",
quantized_model="out/model.quantized.onnx",
use_synthetic=True,
input_shape=(32, 3, 640, 640), # image batch size set to 32
)
```
| Argument | Default | Purpose |
| --------------- | ----------------------- | ------------------------------------------------------------------------------------ |
| `use_synthetic` | `False` | Generate input instead of taking `eval_data` |
| `input_shape` | inferred from the model | Override when the graph has dynamic axes |
| `num_samples` | `5` | How many distinct input tensors of shape `input_shape` to generate and cycle through |
## Terminal output
With `verbose=True` (the default) the benchmark prints each run's latency and tracks the running average:
```text theme={null}
[benchmark] plan: 2 models x (5 warmup + 20 timed runs) over 12 provided sample(s) of shape (1, 3, 224, 224) on arm. Press Ctrl+C to stop early.
[benchmark] quantized: warming up (5 runs)...
[benchmark] quantized: run 12/20 · sample 12/12 · last 4.31 ms · avg 4.52 ms
...
[benchmark] summary
original: 9.14 ms/inference (20 runs, CPUExecutionProvider)
quantized: 4.48 ms/inference (20 runs, CPUExecutionProvider)
speedup: 2.04x
hardware: arm (Darwin arm64)
input: (1, 3, 224, 224) x 12 provided samples
```
## The output is a `BenchmarkResult`
| Field | Meaning |
| ---------------------------------------------- | ----------------------------------------------------------------------------------- |
| `latency_ms_original` / `latency_ms_quantized` | Mean per-inference latency in ms |
| `latency_speedup` | original / quantized |
| `metadata["hardware"]` | CPU, GPU (when present), OS, architecture |
| `metadata["provider_quantized"]` / `_original` | ONNX Runtime execution provider actually used |
| `metadata["input_shape"]` | Shape the models were actually fed |
| `metadata["input_source"]` | `"eval_data"` or `"synthetic"` |
| `metadata["num_samples"]` | Number of samples cycled through (`len(eval_dat)`, or `num_samples` when synthetic) |
| `metadata["runs_completed_*"]` | Timed runs completed per model |
| `metadata["early_stopped"]` | Whether the run was interrupted |
Benchmark on hardware that matches your deployment target.
## Accuracy is a separate job
The `benchmark` method does not report accuracy. To sign off on a quantized model, please run your task metric on a held-out labeled set, with the same preprocessing step.
## Next steps
Build a representative image set for ptq\_static.
Pick the right PTQ method or QAT.