Calibration

By default, standard preprocessing is applied: the image is decoded as RGB, resized to the graph’s input height and width, scaled to [0,1], and transposed from HWC to CHW. Channel order is not changed, so a model trained on BGR input (the OpenCV default) needs its channels swapped before it reaches us. For custom preprocessing, provide input arrays with the correct dimensions instead. For an example, see Providing calibration data. Calibration images tell the engine what inputs your model typically sees, so it can produce a well-tuned quantized model. Two paradigms use them, for different reasons.

Which methods need calibration

Method	Calibration	Why
PTQ `ptq_dynamic`	No	No representative data required
PTQ `ptq_static`	Yes	The engine calibrates on your samples, then produces the quantized model; faster inference
QAT	Yes	Calibrates and also trains the model on your images; faster inference, further decrease the accuracy drop

PTQ static INT8 or INT4: measurement only

Static INT8 or INT4 needs to know the range of values your activations take, which depends on the inputs the model sees. You provide representative images; the engine calibrates on your samples, then produces the quantized model.

PTQ calibration is not training. It is a set of forward passes that complete quickly, with no gradient updates.

from r3alai.quant import QuantConfig, Quantizer

result = Quantizer(QuantConfig(method="ptq_static")).quantize(
    "model.onnx",
    calibration_data=[
        "/data/images/sample1.jpg",
        "/data/images/sample2.jpg",
    ],
    max_calib_samples=100,
)

Choosing the clipping threshold

Calibration measures the range of values each activation takes, and INT8 or INT4 spends its 256 levels on that range. The question every calibration method answers is where the range should stop: values beyond the threshold are clipped, values inside it get finer resolution. One rare outlier batch can stretch the range far past where the real signal lives, and every level spent covering that gap is a level not spent on the values your model actually sees. calibration_method picks how the threshold is chosen. It applies to ptq_static only.

`calibration_method`	Threshold	Cost	Use when
`minmax` (default)	The largest absolute value observed. Nothing is clipped	Cheapest	The activations have no long tails, or you have few calibration images
`percentile`	The `calibration_percentile` percentile of the observed values (99.99 or 99.999)	One extra pass over the histogram	Activations have outliers you are willing to clip
`entropy`	The threshold whose quantized distribution loses the least information (minimum KL divergence) against full precision	Slowest, seconds per model	You want the threshold picked for you rather than tuned by hand

from r3alai.quant import QuantConfig, Quantizer

config = QuantConfig(
    method="ptq_static",
    calibration_method="percentile",
    calibration_percentile=99.99,
)

result = Quantizer(config).quantize(
    "model.onnx",
    calibration_data=image_paths,
    max_calib_samples=256,
    output_dir="./out",
)

calibration_percentile is a percentile in percent (99.99), not a fraction (0.9999). Passing a fraction raises UnsupportedConfigError instead of silently clipping away most of your distribution. The QAT equivalent, qat_calib_q, is a fraction.

Entropy calibration follows the scheme TensorRT uses: histogram the observed magnitudes into calibration_num_bins bins (2048 by default), then score each candidate threshold by the KL divergence between the full-precision distribution and what INT8 or INT4 can represent below it. It usually lands close to a well-chosen percentile, which is why percentile is the cheaper way to the same place when you know your data. calibration_symmetric forces the range to be symmetric around zero. Leave it off unless you know you need it: activations are quantized to unsigned INT8 or INT4 with a zero point, so a one-sided range (a post-ReLU tensor spanning [0, alpha]) keeps all 256 or 16 levels, while a symmetric range spends half of them below zero where the tensor never goes. Which method was used, with its parameters and the number of samples it saw, is recorded in the deliverable’s manifest.

How many images

All three methods need representative data, and the histogram methods need the most: percentile and entropy estimate a whole distribution per tensor rather than a running maximum. Aim for 256 to 512 diverse samples for those, and note that max_calib_samples caps how many are used (it defaults to 100). The SDK warns if a histogram method is calibrating on fewer than 256 samples.

QAT: calibration + training

QAT uses the same calibration_data, but the engine both calibrates on it and trains the model on it over epochs:

from r3alai.quant import QuantConfig, Quantizer

result = Quantizer(QuantConfig(mode="qat")).train_qat(
    "model.onnx",
    calibration_data=["/data/images/img1.jpg", "/data/images/img2.jpg"],
    epochs=3,
)

Calibration sets each layer’s initial clip threshold before training starts, and training then tunes those thresholds along with the weights. The same three options are available through qat_calib_method: percentile (the default, at quantile qat_calib_q), max, and entropy (over a qat_calib_num_bins histogram).

config = QuantConfig(mode="qat", qat_calib_method="entropy", qat_calib_batches=32)

For QAT, calibration images also serve as training data. Use more images and more epochs than a PTQ smoke test.

Providing calibration data

Pass image file paths (the SDK uploads them with your model) or preprocessed arrays:

import numpy as np
from r3alai.quant import QuantConfig, Quantizer

calib = [preprocess(p) for p in image_paths]

Quantizer(QuantConfig(method="ptq_static")).quantize(
    "model.onnx",
    calibration_data=calib,
)

Building a good calibration set

Match production preprocessing

Same resolution, normalization, and letterboxing as your inference pipeline.

Cover the deployment distribution

256-1024 images spanning lighting, object sizes, scene types, and edge cases. Fewer works for minmax, but the histogram methods get noisy below roughly 256.

Validate afterwards

Compare quantized vs original outputs. See Benchmarking. Especially

Get Started

Core Concepts

Guides

Which methods need calibration

PTQ static INT8 or INT4: measurement only

Choosing the clipping threshold

How many images

QAT: calibration + training

Providing calibration data

Building a good calibration set

​Which methods need calibration

​PTQ static INT8 or INT4: measurement only

​Choosing the clipping threshold

​How many images

​QAT: calibration + training

​Providing calibration data

​Building a good calibration set

Which methods need calibration

PTQ static INT8 or INT4: measurement only

Choosing the clipping threshold

How many images

QAT: calibration + training

Providing calibration data

Building a good calibration set