Quantization is a technique for reducing the memory and compute needed to run an AI model by representing its weights with lower-precision numbers, for example converting 16-bit floating point weights to 8-bit or 4-bit integers. This shrinks model size and speeds up inference, often with only a small drop in quality, which is what makes it possible to run large open-weight models on consumer GPUs or even laptops. Common formats include GPTQ, AWQ, and GGUF (used by llama.cpp and Ollama). Quantization is central to local and on-device AI: a model that needs 140GB of memory at full precision can often fit in under 40GB when quantized to 4-bit.
Frequently Asked Questions
What is quantization in AI?
Quantization stores a model's weights at lower numerical precision (for example 4-bit instead of 16-bit) to shrink its size and speed up inference, usually with only a small loss in quality.
Why does quantization matter?
It lets large models run on cheaper hardware, including consumer GPUs and laptops, which is why local AI tools like Ollama rely on quantized formats such as GGUF.