Quantization
Storing a model's numbers with less precision so it fits in less memory and runs faster — usually at a surprisingly small cost in quality.
Reviewed July 10, 2026Stable
When not to use it
- When you have the memory. If the full model fits and latency is fine, quantizing buys you nothing and costs some quality.
- On tasks sensitive to numerical precision. Code generation and arithmetic degrade earlier than conversation, and the drop won't show in a perplexity score.
- Without task-specific evaluation. Benchmarks average over exactly the failures you'll notice, and "the numbers look fine" is not the same as "it works."
Reach for something else instead
- A smaller model — often a well-trained small model beats a heavily quantized large one at the same memory budget, and that comparison is rarely run.
- Distillation — train a small model to imitate the big one. More work up front, better quality at size.
- A hosted API if the point was cost rather than privacy or offline operation. Do the arithmetic before buying a GPU.
Read more on the blog
- How quantization shrinks AI models without breaking themA 70-billion-parameter model needs about 140 GB of memory at full precision. Your laptop has 16. Quantization is how the model fits anyway, by storing each weight in far fewer bits, and the surprising part is that you can throw away most of that precision and the model barely notices. Here is why, and where it finally breaks.
- Run an LLM locally: how much VRAM do you need?Whether a model runs on your machine is not a mystery. It is one line of arithmetic: parameters times bytes per parameter, plus the context cache almost everyone forgets. This guide gives you the formula, the numbers for every common model size, and the two traps that cause most out-of-memory errors.
- What is fine-tuning? How to specialize an AI modelFine-tuning takes a general-purpose model and adapts it to your specific task by continuing its training on your examples. It is one of the three ways to shape a model, the most powerful and the most misused. Here is what it actually does, how LoRA made it cheap, and the honest answer to whether you should do it.
- Knowledge distillation: how small models learn from big onesThe small, fast AI models you run on a laptop or serve cheaply to millions of users were often not just shrunk from big models. They were taught by them. Knowledge distillation trains a compact student model to mimic a large teacher, and the surprising part is that the student learns more from the teacher's uncertainty than from the raw right answers.
Further reading
- Dettmers et al. (2022), LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — the outlier problem, and the paper that made 8-bit routine.
- Frantar et al. (2022), GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — the method behind most 4-bit models you'll download.
- Dettmers et al. (2023), QLoRA: Efficient Finetuning of Quantized LLMs — fine-tuning on top of a quantized model, on one GPU.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading unchanged perplexity as unchanged capability. It's an average; the specific things you care about can degrade underneath it.
- Assuming quantization loss is uniform. It isn't — reasoning, code, and long-context work suffer disproportionately.
- Skipping the comparison against a smaller unquantized model at the same footprint. Sometimes it wins, and nobody checks.
At a glance
FieldLanguage & LLMs
Core ideafewer bits per number
8-bitnearly free
4-bitsmall cost, common default
Below 4-bittask-dependent, measure it
DifficultyIntermediate
Often compared with
Quantization vs. distillation — compressing a finished model vs. training a smaller one to imitate it.
Where this sits
24 concepts come first. Understanding it opens up 2 more.
11Levelsteps in
24Needs firstconcepts
2Opens up1% of 310
2Areascrosses fields
Learn these firstLarge Language Model
Computed from the prerequisite graph, not assigned. How this works