How quantization shrinks AI models without breaking them

A 70-billion-parameter model stored at full precision needs about 140 GB of memory just for its weights. A high-end datacentre GPU ships with 80 GB. A good laptop has 16. By that arithmetic, running a serious model on your own hardware should be impossible, and yet people do it every day through tools like Ollama and llama.cpp. The thing that makes it possible is quantization, and it rests on a fact that sounds like it should not be true: you can throw away most of the numerical precision in a model's weights, and the model barely gets worse.

This piece explains how that works. What quantization actually does to a model, why discarding three-quarters of the bits in each weight does so little damage, why it also makes inference faster rather than just smaller, the counterintuitive rule it implies for choosing a model, and the specific place where it stops being free and starts to hurt. Quantization sits alongside mixture of experts and the memory-bound nature of inference as one of the three ideas that determine what running a model actually costs, and it is the one you are most likely to use yourself.

The core idea: fewer bits per weight

A neural network is, concretely, a huge pile of numbers called weights. By default each weight is stored as a floating-point number using 32 or 16 bits, which can represent a value to fine precision. Quantization stores each weight using fewer bits: 8 bits, or 4, or in aggressive cases fewer. That is the whole idea. You take a weight that was a 16-bit float and approximate it with, say, a 4-bit integer, one of only sixteen possible values, chosen to sit as close as possible to the original.

The payoff is directly proportional. Going from 16-bit to 8-bit halves the memory the weights occupy; going to 4-bit quarters it. A 7-billion-parameter model that takes about 14 GB at 16-bit precision drops to around 3.5 GB at 4-bit, which is the difference between not fitting on a laptop and fitting comfortably. This is why quantization is the single technique most responsible for the local-model boom: it turns models that needed a server into models that run on the hardware you already own.

The obvious objection is that this should wreck the model. You are replacing precise numbers with coarse approximations, introducing rounding error into billions of weights. Intuitively that sounds destructive. It mostly is not, and understanding why is the interesting part.

Why throwing away precision barely hurts

The reason quantization works is a property of how neural networks store what they know. A model's knowledge is not held in the exact value of any individual weight; it is spread redundantly across billions of them, in the overall pattern of their interactions. No single weight is load-bearing in a way that its third decimal place matters. So when you round each weight to a nearby coarse value, you introduce a small amount of noise into each one, and the network, which is massively overparameterised and was trained to be robust to noise anyway, simply absorbs it. The errors are small, they are roughly random rather than systematic, and they partially cancel across the many weights feeding into each computation. The model's behaviour shifts a little, but its competence is preserved because that competence never depended on precision to begin with.

There is a useful way to feel this. A weight's exact value is like the exact wording of one sentence in a long report: change a word here and there throughout and the report still says the same thing, because the meaning lives in the whole, not in any single phrase. Quantization is a global, gentle rewording. It would be fatal only if meaning were concentrated in a few exact values, and in a large network it is not, which is precisely why bigger models tolerate quantization better than small ones: they have more redundancy to spare.

Why it also makes inference faster

Quantization is usually introduced as a way to save memory, but it does something else that matters just as much, and it connects directly to how inference actually works. As covered in the piece on why inference is memory-bound, generating each token requires reading the entire set of model weights from memory, and that data movement, not the arithmetic, is the bottleneck. If each weight is 4 bits instead of 16, there is a quarter as much data to move per token. So quantization does not just make the model fit; it makes it generate faster, because the memory bus, which was the limiting resource, now has far less to carry.

It shrinks the other memory hog too. The KV cache, which grows with context length and can rival the weights in size, can itself be quantized, further cutting the data moved per token and freeing room for longer context or larger batches. Between smaller weights and a smaller cache, quantization attacks exactly the bottleneck that governs inference cost. This is why quantized models are not just more portable but cheaper and faster to serve, and why low precision is now built into the hardware itself, with recent accelerators offering native 8-bit and even 4-bit number formats.

The counterintuitive rule: bigger and coarser beats smaller and precise

Put those facts together and they imply a rule that surprises most people the first time they hear it, and that is worth stating plainly because it is the practical heart of the topic. A larger model at lower precision almost always beats a smaller model at higher precision, for the same memory budget. A 70-billion-parameter model quantized to 4 bits will comfortably outperform a 7-billion-parameter model at full 16-bit precision, even though they occupy roughly similar space, because the larger model has vastly more capacity and quantization costs it very little.

The practical guidance that falls out of this is blunt: pick the biggest model your memory can hold, and then quantize it down to fit, rather than picking a small model and running it at full precision. Capacity buys you more than precision does. This is the opposite of the intuition that says "run the model properly," and it is one of the most useful things to internalise about deploying LLMs locally.

Where it breaks: the outlier problem

None of this means quantization is free, and an honest account has to be precise about where it hurts, because that is where all the real engineering lives. The central difficulty is outliers. Although most of a model's weights and activations sit in a modest range, a small fraction, often around one percent of channels, have very large magnitudes. These outliers are a problem because quantization has to map a whole range of values onto a few coarse levels, and a handful of extreme values stretch that range enormously. When the range is stretched to cover the outliers, all the ordinary values in between get squeezed into just a few levels, losing the resolution that mattered. Naive uniform quantization, applied without care for outliers, is what actually breaks a model.

Every serious quantization method is, at bottom, a way of handling outliers. GPTQ quantizes a layer's weights one at a time and, after each, adjusts the remaining weights to compensate for the error just introduced, using information about the weight distribution. AWQ observes that not all weights matter equally, identifies the roughly one percent of channels most important to the model's outputs, and protects them while aggressively quantizing the rest. Other methods isolate the outlier dimensions into higher precision, or apply a mathematical rotation that spreads the outliers out so no single channel dominates the range. The details differ, but the goal is shared: keep the extreme values from destroying the resolution of everything else. It is also why weights are the easy part and activations are the hard part, since activations vary with every input and their outliers are less predictable, which is why many practical schemes quantize weights to 4 bits while keeping activations at 16.

PTQ versus QAT, and the quality ladder

There are two moments you can quantize a model, and the distinction matters. Post-training quantization (PTQ) takes a finished, already-trained model and compresses it after the fact, usually with a small calibration dataset used to set the value ranges well. It requires no retraining, which makes it the only option for the public model checkpoints most people use, and it works well down to about 4 bits. Quantization-aware training (QAT) instead simulates quantization during training, so the model learns weights that are robust to it from the start. QAT is more work and needs the original training setup, but it wins at the lowest bit-widths, below 4 bits, which is why companies shipping models for on-device use increasingly train them quantization-aware.

However you get there, quality degrades gradually as precision drops, and knowing the shape of that curve is what separates good deployment from bad. Eight-bit quantization is nearly lossless; the difference from full precision is hard to detect. Four-bit, using a good scheme, is still strong for general use, and in ordinary conversation most people cannot tell a well-quantized 4-bit model from the original. But the loss is not evenly distributed across tasks: it shows up first and worst on the demanding ones, math, code generation, and multi-step reasoning, where small errors compound and precision matters more. Below 4 bits, at 3 or 2, degradation becomes clearly noticeable even in casual use. The rule of thumb: 8-bit when you want maximum fidelity, 4-bit as the standard sweet spot for local use, and avoid going lower for reasoning-heavy work. (This same quantized-base idea powers efficient fine-tuning: as covered in the fine-tuning piece, QLoRA trains small adapters on top of a 4-bit frozen model, which is what lets people fine-tune large models on a single consumer GPU.)

The short version

Quantization stores a model's weights in fewer bits, 8 or 4 instead of 16, which shrinks the model proportionally: a 7B model drops from about 14 GB to 3.5 GB at 4-bit. It barely hurts quality because a model's knowledge is spread redundantly across billions of weights rather than held in any weight's exact value, so rounding each one introduces small noise the network absorbs. It also speeds up inference, because generating a token is bottlenecked by moving weights from memory, and fewer bits means less data to move. The practical rule is that a bigger model quantized down beats a smaller model at full precision for the same memory. The hard part is outliers, the roughly one percent of extreme values that ruin quantization if handled naively, which is what methods like GPTQ and AWQ exist to manage, and quality falls off fastest on math, code, and reasoning as precision drops below 4 bits.

The idea to hold onto is that a model's competence lives in the pattern of its weights, not their exact values, so you can store them coarsely and lose almost nothing, which is why a big model squeezed into 4 bits beats a small one kept precise. Precision turns out to be the cheapest thing to give up in a neural network, and quantization is the technique that cashes that in, turning models that needed a datacentre into ones that run on the machine in front of you.

Common questions

What is quantization in machine learning? Quantization is the technique of storing a model's weights using fewer bits than the 16 or 32 normally used, for example 8-bit or 4-bit integers instead of 16-bit floats. Each weight is approximated by the nearest of a small set of values. This shrinks the model proportionally: a 7-billion-parameter model drops from about 14 GB to roughly 3.5 GB at 4-bit. It is the main technique that lets large models run on consumer hardware like laptops, and it also speeds up inference by reducing how much data must be moved from memory per token.

Why doesn't quantization ruin the model? Because a model's knowledge is spread redundantly across billions of weights, not stored in the exact value of any single one. Rounding each weight to a coarser value introduces a small amount of roughly random noise, and a large, overparameterised network absorbs it, the errors partly cancel across the many weights feeding each computation. Competence never depended on fine precision, so removing it costs little. This also explains why bigger models tolerate quantization better than small ones: they have more redundancy to spare.

Does quantization make models faster or just smaller? Both. It obviously makes them smaller, but it also makes inference faster, because generating each token is limited by how fast weights can be read from memory rather than by computation. If each weight is 4 bits instead of 16, there is a quarter as much data to move per token, so generation speeds up. Quantizing the KV cache reduces data movement further. Because quantization attacks the memory-bandwidth bottleneck directly, quantized models are cheaper and faster to serve, not just more portable.

Is a bigger quantized model better than a smaller full-precision one? Usually yes, for the same memory budget. A 70-billion-parameter model quantized to 4 bits generally outperforms a 7-billion-parameter model at full 16-bit precision, even at similar file sizes, because the larger model has far more capacity and quantization costs it very little. The practical rule is to choose the biggest model your memory can hold and then quantize it to fit, rather than running a small model at full precision. Capacity buys more than precision does.

What is the outlier problem in quantization? Most of a model's values sit in a modest range, but about one percent of channels have very large magnitudes, called outliers. They are a problem because quantization maps a range of values onto a few coarse levels, and extreme outliers stretch that range so much that ordinary values get squeezed into too few levels, losing resolution. Handling outliers is the central challenge of quantization. Methods like GPTQ (which compensates for error layer by layer) and AWQ (which protects the most important one percent of channels) exist specifically to keep outliers from destroying quality.

What is the difference between PTQ and QAT? Post-training quantization (PTQ) compresses an already-trained model after the fact, using a small calibration dataset to set value ranges. It needs no retraining and is the only option for public model checkpoints, working well down to about 4 bits. Quantization-aware training (QAT) simulates quantization during training so the model learns weights robust to it, which requires the original training pipeline but achieves better quality at very low bit-widths (below 4 bits). PTQ is what most people use on downloaded models; QAT is used by companies preparing models for on-device deployment.

How much quality do you lose at 4-bit versus 8-bit? Eight-bit is nearly lossless; the difference from full precision is very hard to detect. Four-bit, with a good scheme, is still strong and in ordinary conversation is usually indistinguishable from the original, which is why it is the standard sweet spot for local use. The catch is that the loss concentrates on demanding tasks, math, code, and multi-step reasoning, where it shows up first. Below 4 bits (3-bit, 2-bit) degradation becomes clearly noticeable even in casual use. Use 8-bit for maximum fidelity, 4-bit as the default, and avoid lower precision for reasoning-heavy work.