What is quantization?

What is Quantization?

Quantization is a model compression technique that reduces the numerical precision of an artificial intelligence model's parameters and activations. Instead of storing and calculating numbers using high-precision formats such as 32-bit (FP32) or 16-bit floating-point (FP16 or BF16), quantization maps those continuous values to lower-bit representations, such as 8-bit integers (INT8), 4-bit numbers (INT4/FP4), or even lower bitwidths. This substantially lowers GPU memory (VRAM) consumption, speeds up memory bandwidth transfers, and lowers inference serving costs.

An everyday analogy is converting a high-resolution photograph with 16.7 million colors (24-bit TrueColor) down to a 256-color palette (8-bit GIF). The compressed image retains virtually the entire composition, shape, and visible detail, but occupies a fraction of the original file size.

How it works

Quantization replaces continuous floating-point values with discrete lower-bit representations through systematic mathematical mapping:

1. Range selection and scaling factor calculation: The quantizer evaluates a tensor's continuous range [val_min, val_max]. It determines a scale factor S and an optional integer zero-point Z that map the float dynamic range onto the target discrete range (such as [-128, 127] for signed INT8 or [-8, 7] for signed INT4):

   • In symmetric quantization, the range is centered around zero (Z = 0), and S = val_max / q_max.
   • In asymmetric quantization (affine mapping), both S and Z are computed to accommodate shifted distributions: S = (val_max - val_min) / (q_max - q_min) and Z = round(q_min - (val_min / S)).

2. Discretization and packing: Floating-point value x is converted to quantized integer q via q = round(x / S + Z), with values clamped to the min/max bounds. For 4-bit representations, two 4-bit integers are typically packed into a single 8-bit byte to save space in storage and system RAM.

3. Granularity and blocking: Mapping an entire weight matrix to a single scale is known as per-tensor quantization. To preserve accuracy across complex layers, modern methods use per-channel or group-wise (block) quantization, where weights are segmented into groups (commonly 128 or 256 parameters per group), each with its own local scale.

4. Handling activation outliers: In large language models, certain hidden state feature dimensions exhibit extreme outlier magnitudes (often exceeding 6.0 while standard weights stay between -3.5 and 3.5). Methods like LLM.int8() isolate outlier channels and compute them in 16-bit floating-point, while quantizing the remaining 99.9% of weights into INT8. Methods like AWQ (Activation-aware Weight Quantization) measure activation magnitudes to protect the top 1% most salient weight channels from precision loss.

5. Dequantization and compute execution: During inference, fused matrix multiplication kernels dequantize integers back to floating point on-the-fly (x ≈ S * (q - Z)) inside GPU registers, or perform direct integer arithmetic (INT8 x INT8 -> INT32) before accumulating into higher precision.

Quantization vs Pruning

Quantization and pruning both reduce neural network size, but they alter model architecture and tensor structures differently:

• Quantization retains every single layer, weight connection, and parameter in the network, but reduces the bit precision used to represent each numerical value (e.g., storing a weight as 4 bits instead of 16 bits).
• Pruning completely removes redundant or less impactful parameters, setting weights to zero (sparse pruning) or cutting entire attention heads and feedforward channels (structured pruning).
• While pruning eliminates connections to reduce the total parameter count, quantization maintains the dense tensor shape and replaces high-precision arithmetic with low-bit representations.

A practical example

The following Python script uses Hugging Face's transformers and bitsandbytes library to load an open-weights model in 4-bit precision (NF4) using BitsAndBytesConfig:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "meta-llama/Llama-3.1-8B-Instruct"

# Configure 4-bit NormalFloat (NF4) quantization
quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# Load quantized model onto GPU
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=quant_config,
    device_map="auto"
)

print(f"Model memory footprint: {model.get_memory_footprint() / (1024**3):.2f} GB")

What the output means

A standard 8-billion parameter model in 16-bit precision (bfloat16) occupies approximately 16.77 GB of VRAM just to hold model weights. Loading the model with load_in_4bit=True and double quantization reduces the weight footprint to roughly 6.44 GB, allowing it to run comfortably on a single consumer GPU with 8 GB to 12 GB of VRAM without requiring multi-GPU partitioning.

Tools and implementations

Tool	What it is for	Link
bitsandbytes	PyTorch library enabling on-the-fly 8-bit (LLM.int8()) and 4-bit (NF4/FP4) inference and QLoRA fine-tuning	github.com/bitsandbytes-foundation/bitsandbytes
AutoGPTQ / GPTQModel	Post-training quantization tool using second-order Hessian error compensation for 4-bit weight compression	github.com/ModelCloud/GPTQModel
AutoAWQ	Efficient activation-aware 4-bit weight quantization optimized for GPU serving in engines like vLLM	github.com/casper-hansen/AutoAWQ
llama.cpp / GGUF	C/C++ inference runtime supporting multi-level block-wise quantization (Q4_K_M, Q8_0) for CPU and edge deployment	github.com/ggml-org/llama.cpp
PyTorch torchao	Native PyTorch architecture and ops library for INT4, INT8, and FP8 quantization and kernel compilation	github.com/pytorch/ao

Where it goes wrong

Quantization offers strong memory reductions, but comes with technical trade-offs:

• The sub-4-bit precision cliff: Quantizing down to INT8 or standard 4-bit (AWQ/GPTQ) typically preserves downstream benchmark accuracy within 1-2% of baseline. However, dropping to 3-bit or 2-bit representations often triggers severe perplexity spikes and reasoning degradation unless extensive quantization-aware training (QAT) is applied.
• Activation outlier sensitivity: Standard uniform quantization fails when features develop extreme numerical outliers. Without outlier preservation techniques (such as mixed precision in LLM.int8() or scaling factors in AWQ), activation dynamic ranges clip, causing catastrophic failure in syntax and instruction following.
• Calibration dataset mismatch: Post-training quantization algorithms (such as GPTQ) optimize weight rounding against a representative calibration dataset (e.g., 128 to 512 token sequences). If calibration samples fail to reflect target domains (e.g., code, math, or multilingual text), downstream accuracy in those areas can degrade significantly.
• Inference overhead without hardware kernels: Loading 4-bit weights saves GPU VRAM, but unless paired with fused dequantization kernels (such as Marlin or TensorRT-LLM kernels) or native low-bit hardware support (like FP8 or NVFP4 tensor cores), unpacking bits on-the-fly can introduce computational latency compared to native 16-bit math.

Why it matters now

As frontier language models scale into hundreds of billions of parameters, memory capacity and memory bandwidth become the primary bottlenecks in production AI infrastructure. Quantization makes serving massive models economically viable by reducing VRAM requirements by 50% to 75%, allowing inference servers to host larger context windows, run higher batch sizes, and lower per-token latency.

FAQs

What is the difference between PTQ and QAT? Post-Training Quantization (PTQ) quantizes an already-trained model in minutes or hours without full retraining, using a small calibration dataset to minimize rounding errors. Quantization-Aware Training (QAT) simulates low-precision quantization during model training or fine-tuning, allowing the network weights to adapt to precision loss and better preserve accuracy at low bitwidths (such as 2-bit or 4-bit).

Does 4-bit quantization reduce model file size on disk? It depends on the storage format. Standard PyTorch checkpoints saved with load_in_4bit=True often retain higher-precision container structures, whereas dedicated formats like GGUF or serialized AWQ/GPTQ safetensors pack weights into packed integer tensors, shrinking disk footprint from ~140 GB down to ~40 GB for a 70B model.

What is the difference between weight-only quantization and weight-activation (W8A8/W4A4) quantization? Weight-only quantization (e.g., W4A16) compresses stored model weights to 4 bits while keeping activations and matrix multiplication in 16-bit floats, saving memory and memory bandwidth. Weight-activation quantization (e.g., W8A8 or W4A4) quantizes both weights and input activations, enabling integer arithmetic (such as INT8 tensor cores) for faster compute throughput.

How does FP8 compare to INT8 quantization? FP8 uses an 8-bit floating-point format (such as E4M3 or E5M2) with dedicated exponent and mantissa bits, offering a wider dynamic range that handles activation distributions better than fixed-scale INT8. Modern GPUs (such as NVIDIA H100 and newer) offer hardware-accelerated FP8 tensor cores that match 16-bit baseline perplexity with minimal calibration overhead.