What is quantization in AI?

Quantization is the process of reducing the numerical precision of a model's weights, making it smaller and faster to run with a typically small impact on accuracy.

Does quantization make a model worse?

It can cause a small drop in accuracy, but well-implemented quantization, especially at 8-bit precision, often has only a minor or imperceptible effect on quality.

Why is quantization important for running AI locally?

Quantization reduces a model's memory and compute requirements enough that large models can run on consumer hardware like laptops and phones instead of requiring data-center infrastructure.

What Is Quantization? — AI Glossary

Quantization

Quantization is a technique that reduces the numerical precision of an AI model's parameters, making it smaller and faster to run with only a small, often negligible, drop in accuracy.

Key takeaways

Quantization reduces the numerical precision of a model's weights to make it smaller and faster to run.
It can shrink model size by 2 to 4 times or more with only a small drop in accuracy.
Quantization is what makes it possible to run large models on consumer hardware like laptops and phones.
Quantized models trade a small amount of precision for significantly lower memory and compute requirements.
Quantization is often combined with distillation, which trains a separate smaller model, to further reduce cost.

What is quantization?

Quantization reduces the precision of the numbers that make up a model's weights, for example converting them from 16-bit or 32-bit floating-point numbers down to 8-bit or even 4-bit representations. This shrinks the model's memory footprint and speeds up the calculations needed to run it, since lower-precision math is cheaper for hardware to compute.

Why quantization matters

A model's weights can number in the billions, and storing and computing with each one at full precision requires substantial memory and computing power. Quantization can shrink a model's size by 2 to 4 times or more, which is often the difference between a model that requires a data center to run and one that can run on a laptop or phone.

Quantization and accuracy

Reducing precision does discard some information, so quantized models can lose a small amount of accuracy compared to the full-precision original. In practice, well-implemented quantization, especially down to 8-bit, often produces only a minor, sometimes imperceptible, quality difference, which is why it's widely used in production rather than being a rare trade-off.

Quantization vs distillation

Quantization and distillation both aim to make models cheaper to run, but they work differently. Quantization keeps the same model and reduces the precision of its existing numbers; distillation trains an entirely new, smaller model to imitate the behavior of a larger one. The two techniques are often combined to get the smallest, fastest model for a given level of quality.

Frequently asked

What is quantization in AI?: Quantization is the process of reducing the numerical precision of a model's weights, making it smaller and faster to run with a typically small impact on accuracy.
Does quantization make a model worse?: It can cause a small drop in accuracy, but well-implemented quantization, especially at 8-bit precision, often has only a minor or imperceptible effect on quality.
Why is quantization important for running AI locally?: Quantization reduces a model's memory and compute requirements enough that large models can run on consumer hardware like laptops and phones instead of requiring data-center infrastructure.