What is quantization?
Quantization is the process of converting a model’s weights, and sometimes its activations, from high-precision numbers to lower-precision ones. Models are trained in 16-bit or 32-bit floating point. Quantization rounds those values to 8-bit floating point (FP8), 8-bit integers (INT8), or 4-bit integers (INT4), with scaling factors that keep the rounded values close to the originals. Each step down halves the memory the weights occupy. A 30-billion-parameter model needs about 60 GB in 16-bit precision, about 30 GB in 8-bit, and about 15 GB in 4-bit.
Lower precision also speeds up inference. Moving weights from GPU memory to the compute units is the main bottleneck when generating tokens, and smaller weights move faster. Recent GPUs have dedicated hardware for 8-bit arithmetic, so FP8 and INT8 gain on both memory and speed.
Types of quantization
| Format | Memory per parameter | Typical quality effect | Common use |
|---|---|---|---|
| FP16 or BF16 | 2 bytes | Reference quality | Training and evaluation baselines |
| FP8 | 1 byte | Nearly indistinguishable on most tasks | Production serving on current GPUs |
| INT8 | 1 byte | Small, usually acceptable loss | Serving on older GPUs and CPUs |
| INT4 | Half a byte | Noticeable loss on reasoning and long outputs | Fitting large models on small hardware |
Why quantization matters
Quantization decides which models fit which hardware, and that decides cost. A hospital network that wants a capable model inside its own data center may have one server with a single 80 GB GPU. In 16-bit precision that server runs a model of about 35 billion parameters with little room left for users. In FP8 it runs the same model with memory to spare for the KV cache and concurrent sessions, or a model twice the size. Quantization also makes small language models practical on laptops and edge devices, where memory is measured in single-digit gigabytes.
What to watch for
- Measure accuracy on the company’s own tasks after quantizing, because losses show up unevenly and hit math, code, and long reasoning first.
- Prefer quantized weights published or validated by the model’s developer over ad hoc conversions.
- Confirm that the inference server and GPU generation support the chosen format, since FP8 requires newer hardware.
- Quantize the KV cache as well when memory is tight, but test the effect on long conversations.