How LLM inference works: why it's bound by memory, not compute

There is a fact about running large language models that surprises almost everyone the first time they meet it: buying a GPU with more raw compute frequently does not make the model generate text any faster. You would expect more floating-point operations per second to mean more tokens per second. It often does not, and understanding why is the key to understanding the economics, latency, and context limits of every AI product you use.

The reason is that generating text with an LLM is not primarily limited by computation. It is limited by memory bandwidth, how fast data can be moved from the GPU's memory to its compute units. This piece explains how inference actually works underneath: the two very different phases a request goes through, the KV cache that quietly dominates the whole process, why generating each token is a memory problem rather than a math problem, and how all of this determines what long context costs and why serving these models is a systems problem rather than a matter of buying a bigger chip. This is the layer beneath the conceptual account in how a sentence becomes an answer: not what the model computes, but what actually makes it slow and expensive.

Two phases with opposite bottlenecks

The first thing to understand is that answering a prompt is not one operation but two, and they have almost nothing in common performance-wise. When your request arrives, the model first has to read your prompt, then generate a response, and these are called prefill and decode.

Prefill processes your entire prompt at once. Because all the input tokens are available together, the model can run them through in parallel, one big batch of matrix multiplications, exactly the kind of dense, parallel work a GPU is built for. Prefill is compute-bound: the GPU's tensor cores are the limiting resource, and they are busy. This phase determines your time to first token, the pause before the answer starts appearing. A prompt of several thousand tokens is crunched in a fraction of a second on a modern accelerator.

Decode is the other phase, and it is fundamentally different. Once the first token is produced, the model generates the rest of the answer one token at a time, autoregressively: each new token depends on all the tokens before it, so they cannot be produced in parallel. The model runs a full forward pass to produce a single token, then does it again for the next, and again, once per token of output. This phase determines the time between tokens, the speed at which words stream out. And decode, unlike prefill, is memory-bound. This asymmetry is the heart of LLM performance.

Why generating each token is a memory problem

Here is the crux, and it is worth going slowly because it is counterintuitive. To generate a single token, the model must read its entire set of weights, tens or hundreds of gigabytes, from the GPU's high-bandwidth memory into the compute units, and then do a relatively small amount of arithmetic with them before producing one token. The ratio of computation to data moved is low. In the language of the field, decode has low arithmetic intensity: you move an enormous amount of data to do a little math.

The consequence is that the bottleneck is not the compute units, which finish their small job quickly and then sit idle, but the memory bus, which is saturated moving weights around. During decode the expensive tensor cores are mostly waiting on memory. This is why a GPU with more compute but the same memory bandwidth barely speeds up generation: you were never compute-limited to begin with. What you need is faster or more efficient memory access, not more math. It is the difference between a chef who can chop instantly but has to walk to a distant pantry for every ingredient. Making the chef's knife faster does nothing; the walk to the pantry is the bottleneck. In decode, loading the weights and the cache is the walk to the pantry.

The KV cache: the thing that actually dominates

The second half of the story is a data structure that most users never hear about but that governs the cost and limits of everything: the KV cache.

Recall from how transformers work that attention has each token look at every previous token, computing a Query for the current token and matching it against a Key and Value for every earlier one. During decode, generating token 500 means attending to the previous 499. The naive approach would recompute the Key and Value vectors for all previous tokens at every single step, which is enormous redundant work, quadratic in the sequence length. The KV cache is the optimisation that fixes this: once a token's Key and Value vectors are computed, they are stored and reused on every future step, so each new token only computes its own K and V and looks up the rest. It trades memory for computation, and it is not optional in practice; without it, generation would be far too slow.

But the trade has a sharp edge. The KV cache grows linearly with the length of the sequence, because every token adds its Key and Value vectors to the store, and it grows with the number of simultaneous requests too. This makes it large. For a big model with a long context, the KV cache can reach many gigabytes, and at reasonable batch sizes it can rival or exceed the size of the model weights themselves. And crucially, during decode that entire growing cache has to be read from memory at every step, on top of the weights. So the KV cache is not just a memory-capacity problem; it directly feeds the memory-bandwidth bottleneck. The longer your context, the bigger the cache, the more data moved per token, the slower and costlier each token becomes.

This single fact explains a lot of observed behaviour. It is why a long conversation gets gradually slower and more expensive as it goes: the KV cache is growing with every exchange. It is why long-context requests cost more than their token count alone suggests. And it is why context length, which sounds like a simple number, is one of the hardest and most expensive things to scale in practice. The quadratic cost of attention often gets the blame, but in production serving, the linear-but-large KV cache and its bandwidth demands are frequently the binding constraint.

Batching: how serving actually gets efficient

If decode is bottlenecked on reading the weights from memory, there is an obvious lever, and it is the one that makes serving economical. When you load the entire set of model weights from memory to generate a token for one user, you can, for almost the same memory cost, generate a token for many users at once, as long as their requests are processed together. This is batching: run many requests through the model simultaneously so the expensive weight-loading is shared across all of them. Batching is what turns a memory-bound workload into an efficient one, and it is why serving many users at once is dramatically cheaper per token than serving one.

But batching has a ceiling, and the ceiling is set by the KV cache. Every request in a batch needs its own KV cache held in memory at the same time, so the more you batch, the more memory the caches consume, until you run out. Batch size is ultimately limited not by compute but by how many KV caches fit in GPU memory alongside the weights. This is the tension at the centre of LLM serving: bigger batches mean better throughput and lower cost per token, but they consume more memory and can worsen latency for any individual user. Much of the engineering of a serving system is managing this trade-off well.

Inference is a systems problem, not a GPU problem

The practical upshot, and the thing worth taking away, is that LLM inference performance is determined less by the raw power of the accelerator than by how cleverly the whole system manages memory. Teams routinely spend more on faster GPUs and are disappointed, because their bottleneck was never compute. The real levers are in software: how the KV cache is stored and reused, how requests are batched, how the scheduler keeps the hardware busy.

This is why the interesting work in inference is largely about memory. PagedAttention borrowed the idea of virtual-memory paging to store the KV cache in non-contiguous blocks, cutting the fragmentation that wasted capacity and letting more requests share memory. FlashAttention restructured the attention computation to move far less data to and from memory. Grouped-query attention shrinks the KV cache by sharing Keys and Values across attention heads. Prefix caching reuses the KV cache for prompts that share a common beginning, so a shared system prompt is not reprocessed for every request. Quantization shrinks both weights and cache so less data moves. Continuous batching lets new requests join a batch mid-flight to keep utilisation high. Every one of these is fundamentally a technique for moving less data or moving it more cleverly, because moving data is the bottleneck.

It is also why reasoning models are expensive in a specific way: they generate very long chains of output tokens, and every one of those tokens is a full memory-bound decode step reading the weights and an ever-growing cache. Thinking longer means more decode steps, which means more of exactly the slow, bandwidth-limited work that dominates inference cost. The economics of the reasoning era are, at bottom, the economics of the decode phase.

The short version

An LLM answers in two phases. Prefill reads your whole prompt in parallel and is compute-bound, setting the delay before the first token. Decode generates the answer one token at a time and is memory-bandwidth-bound, setting the speed tokens stream out. Decode is slow not because the math is hard but because each token requires reading the entire model weights, and the growing KV cache, from memory while doing little computation, so the memory bus is the bottleneck and faster compute barely helps. The KV cache stores past tokens' attention vectors to avoid recomputation, but it grows with context length and batch size until it rivals the weights, which is why long context and long conversations get slower and costlier. Batching shares the weight-loading cost across many requests to make serving efficient, but the KV cache caps how large a batch can be.

The idea to hold onto is that generating text with an LLM is bottlenecked by moving data, not by doing math, so inference is a memory problem, and the KV cache, not the model's raw size, is what usually governs its speed, cost, and context limits. Once you see that decode is a walk to the pantry rather than a chop of the knife, the whole strange economics of running these models, why context is expensive, why batching matters, why a faster GPU disappoints, stops being mysterious and starts being predictable.

Common questions

What is the difference between prefill and decode in LLM inference? They are the two phases of answering a prompt. Prefill processes your entire input prompt at once, in parallel, which is compute-bound work a GPU handles well, and it determines the time to the first token. Decode then generates the response one token at a time, autoregressively, since each token depends on the previous ones, and it is memory-bandwidth-bound, determining how fast tokens stream out. They have opposite bottlenecks, which is why they are optimised very differently and why overall inference performance depends on both.

Why is LLM inference memory-bound rather than compute-bound? Because generating each token requires reading the model's entire set of weights (and the KV cache) from memory while doing only a small amount of computation with them. This low ratio of math to data moved, called low arithmetic intensity, means the compute units finish quickly and sit idle while the memory bus is saturated moving data. The limiting resource is memory bandwidth, not compute. This is why a GPU with more raw compute but the same memory bandwidth barely speeds up token generation.

What is the KV cache? The KV cache stores the Key and Value vectors that attention computes for each token, so they can be reused on later generation steps instead of being recomputed. Without it, generating each new token would require recomputing the attention vectors for all previous tokens every time, which is quadratically expensive. The KV cache trades memory for computation and is essential for practical inference speed. Its downside is that it grows with sequence length and batch size, and it must be read from memory on every decode step, which makes it a major driver of cost and latency.

Why do long conversations and long contexts cost more? Because the KV cache grows linearly with the number of tokens. Every token in the context adds its Key and Value vectors to the cache, so a longer conversation or a longer prompt means a larger cache, which consumes more memory and, more importantly, must be read from memory on every single token generation step. Since decode is memory-bandwidth-bound, a bigger cache directly slows generation and raises cost. This is a large part of why scaling context length is one of the hardest and most expensive problems in serving LLMs.

Why does batching make LLM serving cheaper? Because the dominant cost during decode is loading the model weights from memory, and once you have loaded them to generate a token for one request, you can generate tokens for many requests in the same pass at almost no extra weight-loading cost. Batching shares that expensive memory traffic across many users, dramatically lowering the cost per token. The limit is memory: each request in the batch needs its own KV cache held simultaneously, so the maximum batch size is set by how many caches fit in GPU memory alongside the weights.

Will a faster GPU make my LLM generate text faster? Often not much, if "faster" means more compute but the same memory bandwidth. Because token generation (decode) is limited by how fast data moves from memory, not by how fast the chip computes, extra compute capacity sits idle waiting on memory. What helps decode is higher memory bandwidth, more memory capacity (to allow larger batches), or software techniques that move less data, such as better KV cache management, quantization, and efficient attention kernels. This is why inference is described as a systems problem rather than simply a hardware one.

Why are reasoning models more expensive to run? Because they generate very long chains of thought before answering, and every one of those output tokens is a separate memory-bound decode step that reads the model weights and the growing KV cache from memory. More thinking means more decode steps, which means more of the slow, bandwidth-limited work that dominates inference cost, and a larger KV cache as the reasoning trace lengthens. The cost of reasoning models is essentially the cost of the decode phase multiplied by how much the model chooses to think.