Why does AI need GPUs? Parallelism and the memory wall

It is worth pausing on how strange the situation is. The hardware that runs modern artificial intelligence was designed to draw video game graphics. Not adapted from it, not inspired by it: the same product line, built for rendering pixels, turned out to be the right machine for a completely unrelated purpose. That is not a marketing story, it is a fact about mathematics, and understanding it explains most of what is otherwise mysterious about AI infrastructure. AI runs on GPUs because neural networks are almost entirely matrix multiplication, the same embarrassingly parallel arithmetic that graphics hardware was already built to do, which was an accident of shape rather than a design for AI, and the more important fact today is that the binding constraint has since moved from arithmetic to memory, so modern inference is limited less by how fast a chip can multiply than by how fast it can fetch the numbers to multiply.

This guide explains the single operation that all neural networks reduce to, why a CPU and a GPU are built on opposite philosophies, how the accident happened, and then the part most explanations skip: why the bottleneck moved to memory bandwidth, and why nearly every important efficiency technique in modern AI turns out to be a different way of moving fewer bytes. That last point ties together several ideas that otherwise look unrelated.

The one operation that matters

Underneath all the sophistication, a neural network does one thing over and over. Each layer takes its inputs, multiplies them by a matrix of learned weights, adds a bias, and passes the result onward. Written out, it is Y = WX + B, repeated through layer after layer. Everything a model appears to know is stored in those weight matrices, and everything it does is the consequence of multiplying numbers by them.

The remarkable part is the volume. A single forward pass through a model with seventy billion parameters requires on the order of a hundred trillion or more floating-point operations, just to produce one token of output. Multiply that by every token in every response, and by every user, and the arithmetic requirement becomes staggering. But there is a second property that matters even more than the volume: these multiplications are largely independent. Computing one element of the output does not require having computed the previous one. They can all be done at the same time.

That independence is the whole reason specialized hardware helps. A problem where the work can be split into many pieces that do not depend on each other is what engineers call embarrassingly parallel, and it is the ideal case for hardware that can do many things at once.

Two opposite design philosophies

A CPU and a GPU are both processors, but they were built to optimize for different things, and the difference explains everything that follows.

A CPU is designed for latency on complex, sequential work. It has a small number of very sophisticated cores, each capable of handling intricate branching logic, unpredictable jumps, and long chains of dependent steps where each operation needs the result of the last. It devotes an enormous share of its silicon to control logic and to a deep hierarchy of caches, all aimed at getting one thread of complicated work done as quickly as possible. That is exactly the right design for running an operating system, a database, or ordinary application code.

A GPU is designed for throughput on simple, repetitive work. Instead of a few complex cores it has thousands of much simpler ones, and it spends its silicon on arithmetic units rather than on control logic. Any individual GPU core is unimpressive and would be poor at running general software. The point is that there are thousands of them, all able to apply the same operation to different pieces of data simultaneously. For a task made of billions of independent multiply-and-add operations, that arrangement is overwhelming, and no realistic cluster of CPUs could match it on time or cost.

The historical accident

Now the accident. GPUs were not built for AI. They were built to render graphics, and rendering has a specific mathematical character: to draw a frame you compute the colour of millions of pixels, and each pixel's calculation is largely independent of the others, involving heavy use of matrix and vector arithmetic to transform geometry and shade surfaces. Millions of independent, arithmetic-heavy, matrix-shaped calculations that can all run at once.

That is the same computational shape as a neural network. Researchers in the late 2000s and early 2010s noticed that the hardware built for pixels happened to be nearly ideal for training neural networks, and adopting it produced speedups large enough to change what was practical. This is not a small footnote in AI history. Deep learning's core ideas had existed for decades and underperformed, and one of the main reasons the field took off when it did is that hardware capable of running those ideas at scale already existed for unrelated commercial reasons. The transformer, and the entire era it launched, was built on chips designed for games.

The specialization that followed

Once the match was recognized, hardware stopped being accidental and became purpose-built. Modern accelerators include tensor cores, dedicated units that perform a small matrix multiplication as a single hardware operation rather than assembling it from many smaller steps. They also increasingly support very low-precision number formats, using eight, six, or even four bits per value instead of sixteen or thirty-two, because neural networks tolerate reduced precision far better than most numerical work, and lower precision means more operations per second and less data to move.

Google took the specialization further with the TPU, a chip designed from the start for the matrix arithmetic of neural networks rather than adapted from graphics. Purpose-built accelerators of this kind can deliver better performance per watt than general GPUs, which matters enormously at scale given AI's energy footprint, and large deployments have reported substantial cost reductions from moving inference onto them. The trade is flexibility: the more a chip is specialized for one shape of computation, the less gracefully it handles workloads that depart from that shape, which is a real consideration in a field where architectures keep changing.

The plot twist: the constraint moved to memory

Here is where most explanations stop and where the interesting part begins. Having established that AI needs enormous arithmetic throughput, you would expect the limiting factor to be arithmetic throughput. For modern inference, it usually is not.

Consider what happens when a model generates a single token. To compute that token, the model's weights have to be read out of memory and fed to the arithmetic units. In the straightforward case, each weight is used for a very small amount of arithmetic and then is finished with. So the machine spends a brief moment multiplying and a much longer moment waiting for the next batch of numbers to arrive from memory. The arithmetic units, which are extraordinarily fast, sit idle waiting to be fed. The workload is memory-bandwidth-bound: its speed is set by how many bytes per second can be moved from memory to the processor, not by how many operations per second the processor can perform.

This is the memory wall, and it is a structural problem rather than a temporary engineering shortfall. Compute capability has grown far faster than memory bandwidth for years, and there are genuine physical difficulties in raising both at once, since the chip area and power spent on one is not available for the other. AI accelerators made this trade deliberately, stripping out much of the elaborate cache hierarchy a CPU uses in order to pack in more arithmetic units, which is why CPUs can actually outperform GPUs on problems that are purely bandwidth-limited. The industry's answer has been high-bandwidth memory, which stacks memory dies vertically right beside the processor die to shorten distances and open thousands of parallel channels, reaching several terabytes per second on current datacentre parts, an order of magnitude or more beyond a CPU. It is a remarkable engineering response, and it has still not made memory stop being the bottleneck.

There is a second, simpler memory constraint alongside bandwidth: capacity. A model's weights have to fit in the accelerator's memory to run efficiently at all, along with the intermediate activations and, during training, the gradients and optimizer state that can multiply the requirement several times over. When a model does not fit, it must be split across multiple devices or partly offloaded, both of which cost performance. In practice, memory capacity often determines which hardware can run a given model, before speed enters the conversation.

Why this explains so much

Once you see that inference is memory-bound, a set of techniques that look unrelated turn out to be the same idea in different clothes. Each is a way of moving fewer bytes, or of extracting more useful work per byte moved.

Batching processes many requests together. Because the model's weights must be read anyway, serving a hundred requests in one pass reads those weights once instead of a hundred times, spreading the expensive part across much more useful output. This is why AI serving is far more efficient at high utilization and why a single isolated request wastes most of the hardware's capability.

Quantization stores each parameter in fewer bits. The usual framing is that it saves memory, which it does, but on a memory-bound workload it also directly increases speed, because halving the bytes per parameter halves the data that must be fetched to produce each token. The speedup is close to proportional, which is why quantization is such an unusually good deal.

Mixture of experts activates only a small fraction of a model's parameters for each token, so only those weights need to be read, cutting the bytes moved per token dramatically while requiring the whole model to remain resident in memory. Seen through this lens it is a trade of memory capacity, which is expensive but purchasable, for memory bandwidth, which is the truly scarce resource.

Speculative decoding has a small fast model propose several tokens which the large model then verifies in a single pass. It only makes sense in a memory-bound world: checking several candidate tokens costs barely more than generating one, because the expensive step, reading the large model's weights, happens once either way. If arithmetic were the bottleneck, the technique would not pay.

The KV cache stores the intermediate attention values from previous tokens so they need not be recomputed, trading memory capacity for avoided work, and it grows with the length of the conversation, which is a large part of why long contexts are expensive to serve.

Four techniques, one underlying logic. When you know the bottleneck is data movement rather than calculation, the entire optimization landscape of modern AI becomes legible.

The part that is not silicon

One last element deserves mention, because hardware discussions tend to focus on chips and miss it. A large share of the dominant position in AI hardware rests on software: the mature programming platform, libraries, and tooling that let researchers use the hardware productively, accumulated over more than a decade and represented by CUDA. Competing silicon can match or exceed on raw specifications and still lose because porting an ecosystem is difficult and risky. This is why alternatives have gained the most traction where a single organization controls the whole stack and can absorb that cost, and it is a reminder that in computing hardware, the moat is frequently not the hardware.

The short version

AI runs on GPUs because neural networks reduce to one operation, matrix multiplication, performed in enormous volume with the individual multiplications largely independent of each other. A CPU is built for latency on complex sequential work, with a few sophisticated cores and deep caches, while a GPU is built for throughput on simple repetitive work, with thousands of simple cores spending their silicon on arithmetic. This suits AI perfectly, and it happened by accident: GPUs were designed to render graphics, which is also millions of independent matrix-shaped calculations, so the hardware for the current AI era existed before anyone built it for AI. Specialization followed, with tensor cores, low-precision number formats, and purpose-built chips such as TPUs. The more important modern fact is that the bottleneck moved. Generating a token requires reading the model's weights out of memory, and the arithmetic per byte fetched is small, so inference is limited by memory bandwidth rather than compute, a structural problem known as the memory wall that high-bandwidth memory has mitigated without solving. Memory capacity is a separate hard limit on what can run at all. This reframes batching, quantization, mixture of experts, and speculative decoding as one family of techniques for moving fewer bytes.

The idea to hold onto is that GPUs won AI by an accident of mathematical shape, because rendering pixels and running neural networks are the same kind of massively parallel arithmetic, but the constraint has since shifted from calculating to fetching, so the useful question about AI hardware is no longer how fast it can multiply but how fast it can feed itself, and almost every major efficiency technique is a different answer to that question.

Common questions

Why does AI need GPUs instead of CPUs? Because neural networks consist of enormous numbers of matrix multiplications that are largely independent of one another, which is exactly the workload a GPU is built for. A CPU has a few sophisticated cores optimized for complex sequential logic and low latency, which suits operating systems and general software. A GPU has thousands of simple cores that apply the same operation to many pieces of data simultaneously, optimizing for throughput. A single forward pass through a large model requires on the order of a hundred trillion operations, and no realistic CPU cluster could deliver that on a sensible timeline or budget, whereas parallel hardware handles it naturally.

Why were graphics cards good for AI? Because rendering graphics and running neural networks are the same computational shape. Drawing a frame means computing millions of pixels whose calculations are largely independent and heavily based on matrix and vector arithmetic. A neural network is billions of independent multiply-and-add operations organized as matrix multiplication. Hardware built to do the first turned out to be nearly ideal for the second, which researchers recognized in the late 2000s and early 2010s. This is a real historical accident with large consequences: one reason deep learning took off when it did is that suitable hardware already existed at scale for unrelated commercial reasons.

What is the memory wall in AI? The memory wall is the growing gap between how fast processors can compute and how fast data can be moved from memory to them. Compute capability has increased far faster than memory bandwidth, and there are physical limits on improving both at once, since silicon area and power spent on arithmetic units are unavailable for memory systems. For AI, the consequence is that generating a token requires reading the model's weights from memory while performing relatively little arithmetic per byte read, so the processor waits on memory rather than the reverse. High-bandwidth memory, which stacks memory beside the processor die for thousands of parallel channels, has reduced the problem without eliminating it.

Is AI inference compute-bound or memory-bound? Usually memory-bound. To produce each token, a model's weights must be streamed out of memory into the arithmetic units, and each weight supports only a small amount of computation before the next data is needed. The arithmetic units, which are extremely fast, end up waiting to be fed, so throughput is set by memory bandwidth rather than by peak operations per second. This is why techniques that reduce data movement, such as quantization and mixture-of-experts routing, produce speedups close to proportional to the bytes they save, and why batching many requests together improves efficiency so dramatically.

Why does batching make AI inference more efficient? Because the expensive step is reading the model's weights from memory, and that cost can be shared. Serving one request requires streaming the entire model's weights to produce a token. Serving a hundred requests together streams those same weights once and produces a hundred tokens' worth of useful output from that single read. The memory traffic stays roughly constant while the useful work multiplies, so efficiency rises sharply with utilization. It is also why an isolated single request leaves most of the hardware's capability idle, and why serving economics depend heavily on keeping accelerators busy.

What is the difference between a GPU and a TPU? A GPU is a general parallel processor originally designed for graphics and later specialized for AI with features like tensor cores, and it remains flexible across many kinds of workload. A TPU is an application-specific chip designed from the start for the matrix arithmetic of neural networks. Purpose-built accelerators can achieve better performance per watt and lower cost at scale, which matters given AI's energy demands, and large deployments have reported significant savings from migrating inference to them. The trade-off is flexibility, since a chip specialized for one computational shape handles unusual or rapidly changing workloads less gracefully than a general-purpose one.

Why is Nvidia so dominant in AI hardware? Partly the silicon and substantially the software. Over more than a decade an extensive ecosystem of programming tools, libraries, and frameworks grew up around one platform, and essentially all AI research and production tooling was built to target it. Competing hardware can match or beat raw specifications and still struggle, because moving an established software stack is expensive and risky, and performance on paper does not translate without mature libraries. Alternatives have made the most progress where a single organization controls the entire stack and can absorb the migration cost internally. In computing hardware, the durable advantage is often the ecosystem rather than the chip.