GPU
The chip that made deep learning possible — thousands of small cores doing the same maths at once, which is exactly what neural networks need.
Reviewed July 11, 2026Stable
When not to use it
- For small models or small data. A CPU is fine, cheaper, and simpler, and plenty of production ML never touches a GPU.
- Buying when you'd rent. Unless utilisation is high and sustained, cloud is cheaper once you count power, depreciation, and your time.
- Buying more compute to fix a utilisation problem. A GPU at 30% has a data pipeline problem, and a faster chip will idle harder.
Reach for something else instead
- CPU inference for small models — genuinely viable, especially quantized, and it removes an entire operational category.
- Cloud GPUs for anything intermittent, which is most workloads.
- Hosted inference APIs if you didn't actually want to operate hardware, which is most teams.
Read more on the blog
- Why does AI need GPUs? Parallelism and the memory wallAI runs on graphics chips because of a historical accident: neural networks turned out to need the same kind of arithmetic that rendering pixels does. But the constraint has since moved, and modern AI hardware is limited less by how fast it can calculate than by how fast it can fetch the numbers to calculate with.
- Fixing the seed does not make it reproducibleTraining the same network fifty times with an identical seed produced almost as much variance as fifty different seeds. Roughly 80% of the spread came from GPU arithmetic, not from randomness anyone controls.
- Who invented deep learning, and why it took so longBackpropagation was invented at least four times before it stuck. The ideas behind deep learning were mostly in place by 1990. What was missing was not insight.
- How much energy does AI use? Training vs inferenceThe energy cost of AI is discussed constantly and measured badly. The per-query figures in circulation are stale and vary by an order of magnitude, the balance has shifted from training to inference, and the constraint that actually bites is not generating electricity but delivering it to a particular building.
Further reading
- Krizhevsky, Sutskever & Hinton (2012), ImageNet Classification with Deep Convolutional Neural Networks — the paper that ran on two consumer GPUs and started this.
- Dao et al. (2022), FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — the memory-movement bottleneck, made concrete.
- Jouppi et al. (2017), In-Datacenter Performance Analysis of a Tensor Processing Unit — what a chip built for this from scratch looks like.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Shopping on raw compute. Memory decides what runs; bandwidth usually decides how fast.
- Assuming inference hardware needs match training. Inference fits in far less, and that's often the entire plan.
- Ignoring utilisation. Idle GPUs are the most expensive thing in the building.
At a glance
FieldTools & Ecosystem
Core ideathousands of cores doing the same maths at once
Decides what runsmemory
Decides how fastbandwidth
Usual bottleneckmoving data, not arithmetic
DifficultyBeginner → Intermediate
Often compared with
GPU vs. CPU — thousands of workers doing one thing vs. a few doing anything.
Where this sits
A starting point. Nothing needs to come before it.
0Levelstarting point
0Needs firstnothing
3Opens up1% of 310
1Areastays here
Computed from the prerequisite graph, not assigned. How this works