Quantization methods

The SDK offers two paradigms: post-training quantization (PTQ) and quantization-aware training (QAT). Both take your model in and return a quantized ONNX model plus a manifest. Both run on the R3AL platform; the SDK uploads your model and downloads the result.

Decision guide

Smallest + fastest, have sample images?    →  PTQ: ptq_static   (default)
No calibration data (MatMul-heavy model)?  →  PTQ: ptq_dynamic
PTQ accuracy not good enough?              →  QAT
Low bit-width (3-4 bit) needed?            →  QAT

PTQ methods

PTQ works on any vision ONNX model: classification, detection, segmentation, pose. No training, so it is the fastest path to a quantized model.

ptq_static (default): smallest and fastest, needs images

Needs: a representative set of calibration images (see Calibration).Benefits: the smallest file and the best latency, including on convolution-heavy backbones. The default and the production path when your calibration images match your deployment distribution.Trade-off: poorly chosen images tune the model for the wrong distribution and cost accuracy. A small accuracy drop versus full precision is normal even with good calibration, so always validate.

from r3alai.quant import Quantizer

result = Quantizer().quantize(
    "model.onnx",
    calibration_data=["img1.jpg", "img2.jpg", "img3.jpg"],  # a handful of representative inputs
    output_dir="./out",
)

calibration_method tunes how the activation ranges are derived from those images (minmax, percentile, or entropy). See Calibration.

ptq_dynamic: no calibration data, MatMul-heavy models

Needs: nothing extra, no calibration data.Benefits: zero setup and a smaller file. The path when you have no calibration data on hand.Trade-off: it quantizes MatMul and Gemm (fully connected) layers, so it suits transformer or MatMul-heavy models. On a convolution-heavy vision backbone it finds nothing to quantize and reports an unsupported config, so reach for ptq_static with calibration data there.

from r3alai.quant import QuantConfig, Quantizer

result = Quantizer(QuantConfig(method="ptq_dynamic")).quantize(
    "model.onnx",
    output_dir="./out",
)

QAT

QAT is a separate paradigm, not a PTQ method. It fine-tunes your model with quantization simulated during training, so the model learns to tolerate lower precision. Needs: calibration/training images and a training run on GPU (epochs). Benefits: recovers accuracy PTQ cannot, especially at low bit widths. Trade-off: it costs a training run and needs representative images.

from r3alai.quant import QuantConfig, Quantizer

config = QuantConfig(mode="qat", qat_wbit=8, qat_abit=8)
result = Quantizer(config).train_qat(
    "model.onnx",
    calibration_data=["/data/img1.jpg", "/data/img2.jpg"],
    output_dir="./qat_out",
    epochs=3,
)

See QAT explained for wbit, abit, and epochs.

Comparison

	`ptq_static` (default)	`ptq_dynamic`	QAT
Paradigm	PTQ	PTQ	QAT
Calibration	Images	No	Images
Training	No	No	Yes
Bits	8	8	3-8 (`wbit`/`abit`)
Smaller file	Yes	Yes	Yes
Best size + speed	Yes	No	Depends
Best for	Default: production, have images	No calibration data, MatMul-heavy	Accuracy at low bits

The engine automatically protects accuracy-sensitive layers, so you set the method and images and let the platform do the rest. Always benchmark on your model before shipping.

Discover methods programmatically

from r3alai.quant import list_methods_local

list_methods_local(scope="any_onnx_vision_model")

Returns metadata for each PTQ method plus the QAT paradigm entry, without a network call.

Get Started

Core Concepts

Guides

Quantization methods

Decision guide

PTQ methods

QAT

Comparison

Discover methods programmatically

​Decision guide

​PTQ methods

​QAT

​Comparison

​Discover methods programmatically

Decision guide

PTQ methods

QAT

Comparison

Discover methods programmatically