Quantization is a technique that reduces the numerical precision of an AI model's parameters, making it smaller and faster to run with only a small, often negligible, drop in accuracy.
Quantization reduces the precision of the numbers that make up a model's weights, for example converting them from 16-bit or 32-bit floating-point numbers down to 8-bit or even 4-bit representations. This shrinks the model's memory footprint and speeds up the calculations needed to run it, since lower-precision math is cheaper for hardware to compute.
A model's weights can number in the billions, and storing and computing with each one at full precision requires substantial memory and computing power. Quantization can shrink a model's size by 2 to 4 times or more, which is often the difference between a model that requires a data center to run and one that can run on a laptop or phone.
Reducing precision does discard some information, so quantized models can lose a small amount of accuracy compared to the full-precision original. In practice, well-implemented quantization, especially down to 8-bit, often produces only a minor, sometimes imperceptible, quality difference, which is why it's widely used in production rather than being a rare trade-off.
Quantization and distillation both aim to make models cheaper to run, but they work differently. Quantization keeps the same model and reduces the precision of its existing numbers; distillation trains an entirely new, smaller model to imitate the behavior of a larger one. The two techniques are often combined to get the smallest, fastest model for a given level of quality.
Mistral has introduced a new AI model with 24 billion parameters that uses 4-bit quantization to run efficiently on consumer-grade hardware, making advanced AI capabilities more accessible.
Multiverse Computing's Quantization-Aware Healing compresses AI models to 4-bit precision while improving their accuracy, enabling smaller, faster models for resource-constrained devices.
Hugging Face has integrated Nunchaku 4-bit Diffusion into its Diffusers library, enabling high-quality AI image generation on consumer hardware. The open-source tool uses 4-bit quantization to dramatically reduce memory and compute requirements, making diffusion models accessible to users without expensive GPUs.
Researchers found that compressing AI models to make them faster can subtly change how they think, even when their answers seem correct. This could affect everything from chatbots to AI assistants, making some reasoning errors harder to detect.
Researchers have developed new open-source methods that outperform LoRA, the most popular AI fine-tuning technique. These approaches could make customizing large language models more accessible and affordable for everyone.