Run an LLM locally: how much VRAM do you need?

You want to run a model on your own machine, for privacy, for cost, or because it is satisfying to own the thing. The first question is always the same: will it fit? Most answers you find are either a vague hardware recommendation or a table with no explanation, and neither tells you how to work it out for a model nobody has written about yet. Whether a model runs on your machine is a memory arithmetic problem you can do in one line: parameters times bytes per parameter gives the weights, the context you use drives a cache that can rival the weights at long context lengths, and quantization is the dial that moves both, which means the real question is not whether your GPU is good enough but what fits in your VRAM and at what quality floor.

This guide gives you the formula, the numbers for every common size, the two things that cause most out-of-memory errors, and the point below which shrinking a model stops being worth it. Do the arithmetic once and you will never need a lookup table again.

The one line of arithmetic

A model's weights are just numbers, and the memory they occupy is the count of those numbers multiplied by the size of each one. That gives the formula that governs everything else:

VRAM needed ≈ parameters (in billions) × bytes per parameter, plus overhead.

Bytes per parameter depends on the precision the weights are stored at. Full precision uses four bytes per parameter. Half precision, the usual format models are released in, uses two. Eight-bit quantization uses one. Four-bit uses roughly half a byte. So the same model can occupy wildly different amounts of memory depending only on how precisely each number is stored.

Work an example. A seven-billion-parameter model at half precision needs about fourteen gigabytes, since seven billion times two bytes is fourteen. At eight-bit it needs about seven. At four-bit, roughly three and a half to four. A seventy-billion-parameter model follows the same rule at ten times the scale: about a hundred and forty gigabytes at half precision, seventy at eight-bit, and around thirty-five to forty at four-bit. The convenient shorthand is two gigabytes per billion parameters at half precision, and about half a gigabyte per billion at four-bit.

Then add overhead. The runtime itself reserves memory before a single weight loads, typically under a gigabyte for the compute context and driver infrastructure, and there are activation buffers during generation. Adding fifteen to twenty-five percent on top of the weight figure is the usual safe margin. If you are downloading a quantized file, the file size on disk is a good floor for what it will need in memory, and you should plan for that plus that margin.

The part everyone forgets: the KV cache

Here is the single most common cause of an out-of-memory error on a machine that "should" have had enough room. The weights are only half the calculation.

When a model generates text, it stores the intermediate attention values for every token already in the conversation so it does not have to recompute them for each new token. This is the KV cache, and unlike the weights, it is not fixed. It grows with the length of your context. Formally it scales with the number of layers, the number of key-value heads, the size of each head, the number of tokens in context, and the bytes used per value. Practically, what matters is that it grows linearly with context length and can become very large.

To make it concrete: for a model in the seven to eight billion range, each thousand tokens of context adds roughly a tenth of a gigabyte at half precision. That sounds trivial until you use a long context. At thirty-two thousand tokens, the cache for such a model can reach around four gigabytes, which is as much as the four-bit weights themselves. A model whose weights are under five gigabytes can therefore need close to nine gigabytes in practice, which is exactly why an eight-gigabyte card that comfortably loads the model still fails partway into a long conversation.

Two things mitigate this. Nearly all recent models use grouped-query attention, which shares key and value projections across query heads and cuts cache size substantially with no quality loss. You get that automatically by using a modern model. Beyond that, most serving software can quantize the cache itself to lower precision, cutting it further, which is the standard technique for fitting a long context onto a small card. The practical rule: budget for the weights, then add anywhere from ten percent to more than double depending on how much context you actually intend to use.

The mixture-of-experts trap

The second trap catches people reading modern model names. Many current open-weight models are mixture-of-experts architectures, advertised with two numbers, such as a thirty-billion-parameter model with three billion active per token. It is tempting to size your hardware against the smaller number. That is the wrong number.

Active parameters determine how much computation happens per token, which is why such a model generates at roughly the speed of a much smaller dense one. But the router can select any expert for any token, so every parameter must be resident in memory the whole time. You size VRAM against the total, and you get the speed of the active count. A thirty-five-billion-parameter model with three billion active still needs memory for thirty-five billion parameters, around twenty-two gigabytes at four-bit, while running as fast as a three-billion model.

That is not a flaw, it is the trade the architecture makes, spending memory capacity to save computation. But it means MoE models are memory-hungry relative to their speed, and the headline "active" figure will mislead you into buying too little VRAM.

The quantization floor

Since quantization is the dial that makes everything fit, the obvious move is to turn it all the way down. There is a limit, and knowing where it sits saves you from a disappointing result.

Down to four-bit, modern quantization methods hold up remarkably well, with quality loss that most users will not notice for ordinary tasks. Below that, degradation accelerates sharply. Pushed to two or three bits, a model becomes noticeably less coherent, and complex reasoning suffers first and most visibly.

This produces a rule worth internalizing: a larger model crushed to very low precision is often worse than a smaller model at reasonable precision, even though it occupies similar memory. A seventy-billion model at two-bit and a thirteen-billion model at eight-bit can take up comparable space, and the smaller one will frequently give better answers. At some point you are destroying the capability you wanted the bigger model for in the first place. When memory is tight, stepping down a size class at good precision usually beats staying big at bad precision.

What fits on what

With the arithmetic above you can size any model, but here is the practical shape of it at four-bit, which is where most local use happens. Around eight gigabytes of VRAM comfortably runs models in the seven-to-eight-billion range at moderate context. Twelve to sixteen gigabytes opens up the mid-teens to roughly thirty billion, depending on context. Twenty-four gigabytes handles around thirty billion comfortably with room for long context, and can reach seventy billion only with aggressive quantization and tight context. Getting a seventy-billion model running properly generally means forty gigabytes or more, which in practice implies multiple cards or a machine with large unified memory. Below about six gigabytes, you are limited to small models and much of the work spills to the CPU.

That spilling deserves a note, because it is the difference between "it runs" and "it is usable." When a model does not fit, most software will offload some layers to system memory and compute them on the CPU. This works, and it is slow: throughput typically drops to a few tokens per second, which is fine for a batch job you leave running and not fine for anything interactive. Partial offload is a legitimate technique for occasionally running something too big, not a substitute for fitting the model. This is the same memory-bound reality that governs AI hardware generally, where moving weights, not multiplying them, sets the pace.

One more practical point: unified-memory machines, where the processor and graphics share a single large pool, behave differently from discrete cards. They can hold much larger models than a typical consumer GPU because the pool is large, while generating more slowly than a high-end discrete card because memory bandwidth is lower. If your goal is running big models at acceptable rather than maximum speed, that trade often favours them.

What to check before you download

A short sequence answers the question for any specific model. Find the total parameter count, and if it is a mixture-of-experts model use the total rather than the active figure. Multiply by bytes per parameter for your intended precision, which is half a byte at four-bit and one byte at eight-bit. Add fifteen to twenty-five percent for runtime overhead. Then estimate your context: for a small model, add roughly a tenth of a gigabyte per thousand tokens you actually plan to use, scaling up with model size, and reduce that if you quantize the cache. Compare the total against your available VRAM, remembering that your display and other applications are already consuming some of it.

If the result is close, you have three levers before giving up: use a smaller context, quantize the KV cache, or step down one model size at better precision. If it is far over, the honest answer is that this model is not for this machine, and a smaller one will serve you better than a heavily offloaded large one. It is also worth remembering that distillation has produced small models that are surprisingly capable, so the gap between what fits and what you need is often smaller than the parameter counts suggest.

The short version

Whether a model fits on your machine is arithmetic, not guesswork. Memory for weights equals parameters times bytes per parameter, where full precision is four bytes, half precision two, eight-bit one, and four-bit about half. So a seven-billion model needs about fourteen gigabytes at half precision and under four at four-bit, while a seventy-billion model needs about a hundred and forty at half precision and thirty-five to forty at four-bit. Add fifteen to twenty-five percent for runtime overhead. The commonly forgotten component is the KV cache, which stores attention values for every token in context and grows with context length: for a seven-to-eight-billion model it adds roughly a tenth of a gigabyte per thousand tokens, so a long context can require as much memory as the quantized weights themselves, which is the usual cause of unexpected out-of-memory errors. Grouped-query attention and cache quantization reduce this substantially. For mixture-of-experts models, size against total parameters rather than active ones, since all weights must be resident even though only a few compute per token. Quantization has a floor: below about four-bit, quality degrades quickly, and a large model at very low precision is often worse than a smaller model at good precision.

The idea to hold onto is that running a model locally is a memory budget problem with three terms, weights, overhead, and a context cache that scales with how much context you use, and since quantization moves the first and cache settings move the third, the question is never simply whether your GPU is good enough but which combination of size, precision, and context fits the memory you have without dropping below the quality floor.

Common questions

How much VRAM do I need to run an LLM locally? Multiply the parameter count in billions by the bytes each parameter takes at your chosen precision, then add fifteen to twenty-five percent overhead and an allowance for context. Bytes per parameter are four at full precision, two at half precision, one at eight-bit, and about half at four-bit. At four-bit, a seven-to-eight-billion model needs roughly four to five gigabytes plus context, so eight gigabytes of VRAM is comfortable. Around twelve to sixteen gigabytes reaches the mid-teens to roughly thirty billion parameters, twenty-four gigabytes handles about thirty billion comfortably, and seventy billion generally needs forty gigabytes or more.

How do you calculate LLM memory requirements? Use three terms. First, weights: parameters in billions times bytes per parameter, which is half a byte at four-bit, one byte at eight-bit, and two bytes at half precision. Second, overhead: add fifteen to twenty-five percent for the runtime, driver context, and activation buffers. Third, the KV cache, which grows with context length and can be estimated at roughly a tenth of a gigabyte per thousand tokens for a seven-to-eight-billion model, scaling with model size. Sum the three and compare against free VRAM, remembering that your display and other applications already use some.

What is the KV cache and why does it use so much memory? The KV cache stores the intermediate attention values for every token already in the context so the model does not recompute them for each new token. Because it holds an entry for every token, it grows linearly with context length rather than staying fixed like the weights. For a model in the seven-to-eight-billion range, roughly a tenth of a gigabyte is added per thousand tokens of context, so a thirty-two-thousand-token conversation can consume around four gigabytes, comparable to the four-bit weights themselves. This is the most common reason a model that loads successfully runs out of memory partway through a long session.

Does quantization hurt model quality? Down to four-bit, modern quantization methods hold up well and the quality difference is not noticeable for most everyday tasks. Below four-bit, degradation accelerates sharply, with complex reasoning and long-form coherence suffering first. The practical consequence is a floor: a very large model crushed to two or three bits is often worse than a smaller model at eight-bit occupying similar memory, because at that point the compression has removed the capability you wanted the larger model for. When memory is tight, dropping one size class at good precision usually beats staying large at poor precision.

How much VRAM does a 70B model need? At half precision, about a hundred and forty gigabytes, which is far beyond any single consumer card. At eight-bit, roughly seventy. At four-bit, around thirty-five to forty gigabytes for the weights alone, before overhead and context. In practice that means forty gigabytes or more of usable memory to run it properly, which typically implies multiple graphics cards or a machine with a large unified memory pool. Running one on a twenty-four gigabyte card is possible only with aggressive quantization and a short context, and usually involves offloading layers to system memory, which slows generation to a few tokens per second.

Why do mixture-of-experts models need more VRAM than their speed suggests? Because the two numbers in their description measure different things. The active parameter count determines how much computation happens per token, so such a model generates at the speed of a much smaller dense model. But the router can select any expert for any token, so all the weights must stay resident in memory at all times. You therefore size memory against the total parameter count and get the speed of the active count. A thirty-five-billion model with three billion active needs memory for thirty-five billion parameters, around twenty-two gigabytes at four-bit, while running roughly as fast as a three-billion model.

Can I run an LLM without a GPU? Yes, but expect a large speed penalty. Models can run entirely on the CPU using system memory, and software commonly supports splitting a model between GPU and CPU when it does not fit in VRAM. The cost is throughput: CPU and offloaded inference typically produce a few tokens per second rather than tens, which is workable for batch jobs you leave running and frustrating for interactive use. If you have any reasonably modern GPU, keeping the model in VRAM matters far more for speed than the processor does, because generation speed is limited by memory bandwidth rather than raw computation.