What is quantization in LLM?
Quantization in an LLM means converting the model's weights from high-precision numbers into lower-precision numbers so the model takes less memory and runs faster.
An LLM is mostly a giant list of decimal numbers called parameters. By default each one is often stored in 16 bits. Quantization rounds those numbers onto a smaller grid of allowed values, for example 256 levels (8-bit) or just 16 levels (4-bit), plus a few scale factors to map them back.
An everyday comparison: a high-quality photo and a compressed WhatsApp version of the same photo. The compressed one is much smaller and still looks fine on your phone, but zoom in and you may notice some detail is gone. Quantization does something similar to a model's numbers.
16-bit vs 8-bit vs 4-bit quantization: what is the difference?
The fewer bits per weight, the smaller and usually faster the model, but the higher the risk of quality loss.
Rough memory needed just for the weights of a 7 billion parameter model:
| Precision | Bytes per weight | Approx. weight memory (7B) | Typical quality effect |
|---|---|---|---|
| 32-bit (FP32) | 4 | about 28 GB | Full precision, mainly used in training |
| 16-bit (FP16 / BF16) | 2 | about 14 GB | Standard for serving, treated as the baseline |
| 8-bit (INT8) | 1 | about 7 GB | Usually very close to 16-bit |
| 4-bit | 0.5 | about 3.5 to 4 GB | Small drop on many tasks, varies by method |
| 2 to 3-bit | under 0.5 | under 3 GB | Often a noticeable drop |
These are estimates for weights only. Actual memory is higher because of the context window cache and software overhead, and quality effects depend on the model, the method and the task.
Does quantization reduce LLM accuracy?
Yes, quantization can reduce accuracy, but at 8-bit the drop is often tiny, and at 4-bit good methods keep most of the original quality for everyday tasks.
Problems show up more in tasks that need precision: long multi-step maths, tricky code, rare languages and very long contexts. Smaller models also tend to suffer more from aggressive quantization than larger ones.
Researchers reduce the damage with smarter methods. Some keep a few especially important values in higher precision, some calibrate the rounding on sample data, and some quantize in small groups with their own scale factors. The LLM.int8 paper, for example, showed that handling a small number of outlier values separately lets large models run in 8-bit with little loss.
I always tell students to test a quantized model on their own prompts before trusting general claims. A 4-bit model that is fine for summarising notes may still struggle with your Python assignment.
How does quantization help run LLMs on a laptop?
Quantization lets a 7B or 8B model fit into roughly 4 to 6 GB of memory, which many modern laptops and some phones can handle.
Without it, the same model needs about 14 GB at 16-bit, which is out of reach for most student laptops. With 4-bit weights, you can download an open-source model and chat with it fully offline. That makes local inference practical for privacy, experiments and learning.
Quantization also matters for fine-tuning. The QLoRA method fine-tunes small adapter layers on top of a frozen 4-bit model, which cut the memory needed so much that the authors fine-tuned a 65B model on a single 48 GB GPU. If you want the background, read what fine-tuning is.
What are the common quantization methods and formats?
The most common approaches are post-training quantization, which converts an already-trained model, and quantization-aware training, which teaches the model to cope with low precision during training.
Names you will see on model download pages include:
- GGUF: a file format used by llama.cpp and tools built on it, popular for running models on CPUs and laptops.
- GPTQ and AWQ: post-training methods that produce 4-bit weights for GPU inference.
- bitsandbytes: a library used with Hugging Face Transformers for 8-bit and 4-bit loading, also used in QLoRA.
Each method has trade-offs in speed, quality and hardware support, and new options appear regularly. Check the Hugging Face quantization docs for the current list supported by your tools.