GPU
The chip that made deep learning possible — thousands of small cores doing the same maths at once, which is exactly what neural networks need.
When not to use it
- For small models or small data. A CPU is fine, cheaper, and simpler, and plenty of production ML never touches a GPU.
- Buying when you'd rent. Unless utilisation is high and sustained, cloud is cheaper once you count power, depreciation, and your time.
- Buying more compute to fix a utilisation problem. A GPU at 30% has a data pipeline problem, and a faster chip will idle harder.
Reach for something else instead
- CPU inference for small models — genuinely viable, especially quantized, and it removes an entire operational category.
- Cloud GPUs for anything intermittent, which is most workloads.
- Hosted inference APIs if you didn't actually want to operate hardware, which is most teams.
Sources & further reading
- Krizhevsky, Sutskever & Hinton (2012), ImageNet Classification with Deep Convolutional Neural Networks — the paper that ran on two consumer GPUs and started this.
- Dao et al. (2022), FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — the memory-movement bottleneck, made concrete.
- Jouppi et al. (2017), In-Datacenter Performance Analysis of a Tensor Processing Unit — what a chip built for this from scratch looks like.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Shopping on raw compute. Memory decides what runs; bandwidth usually decides how fast.
- Assuming inference hardware needs match training. Inference fits in far less, and that's often the entire plan.
- Ignoring utilisation. Idle GPUs are the most expensive thing in the building.
At a glance
FieldTools & Ecosystem
Core ideathousands of cores doing the same maths at once
Decides what runsmemory
Decides how fastbandwidth
Usual bottleneckmoving data, not arithmetic
DifficultyBeginner → Intermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
GPU vs. CPU — thousands of workers doing one thing vs. a few doing anything.