Quantization
Storing a model's numbers with less precision so it fits in less memory and runs faster — usually at a surprisingly small cost in quality.
When not to use it
- When you have the memory. If the full model fits and latency is fine, quantizing buys you nothing and costs some quality.
- On tasks sensitive to numerical precision. Code generation and arithmetic degrade earlier than conversation, and the drop won't show in a perplexity score.
- Without task-specific evaluation. Benchmarks average over exactly the failures you'll notice, and "the numbers look fine" is not the same as "it works."
Reach for something else instead
- A smaller model — often a well-trained small model beats a heavily quantized large one at the same memory budget, and that comparison is rarely run.
- Distillation — train a small model to imitate the big one. More work up front, better quality at size.
- A hosted API if the point was cost rather than privacy or offline operation. Do the arithmetic before buying a GPU.
Sources & further reading
- Dettmers et al. (2022), LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — the outlier problem, and the paper that made 8-bit routine.
- Frantar et al. (2022), GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — the method behind most 4-bit models you'll download.
- Dettmers et al. (2023), QLoRA: Efficient Finetuning of Quantized LLMs — fine-tuning on top of a quantized model, on one GPU.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading unchanged perplexity as unchanged capability. It's an average; the specific things you care about can degrade underneath it.
- Assuming quantization loss is uniform. It isn't — reasoning, code, and long-context work suffer disproportionately.
- Skipping the comparison against a smaller unquantized model at the same footprint. Sometimes it wins, and nobody checks.
At a glance
FieldLanguage & LLMs
Core ideafewer bits per number
8-bitnearly free
4-bitsmall cost, common default
Below 4-bittask-dependent, measure it
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Quantization vs. distillation — compressing a finished model vs. training a smaller one to imitate it.