TPU
Google's chip built only for neural networks — the case that specialisation beats general-purpose hardware, and the only serious alternative to NVIDIA.
When not to use it
- With dynamic shapes. XLA needs static shapes and recompiles when they change. That'll dominate your runtime.
- With small batches. The array needs feeding. Underfed, you've bought nothing.
- When you need custom kernels. The CUDA ecosystem is where that flexibility lives.
- In PyTorch-native workflows. It works and it isn't the path of least resistance.
Reach for something else instead
- GPUs + CUDA — worse per watt, and the ecosystem is eighteen years deep.
- Inference-specific accelerators — serving is more standardised; the moat is thinner there.
- CPUs — for small models, still fine, and people forget.
Sources & further reading
- Jouppi et al. (2017), In-Datacenter Performance Analysis of a Tensor Processing Unit — ISCA; unusually candid, including that it was memory-bound too.
- Kung (1982), Why Systolic Architectures? — the idea, thirty years before it mattered.
- Wang, Choi et al. (2019), bfloat16 and mixed-precision training — range beats precision, and the format everyone adopted.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming specialisation escaped the memory wall. Jouppi's own paper says it didn't.
- Using small or variable batches. You've bought a systolic array and starved it.
- Thinking anyone can copy this. It exists because Google buys its own chips for its own workload.
- Missing that bfloat16 was the durable contribution. It's in every vendor's silicon now.
At a glance
FieldTools & Ecosystem
The questionif you only run neural networks, what's the chip?
The answera systolic array; weights stay put, data flows through, memory traffic mostly eliminated
Reported gains15–30× performance, 30–80× per watt vs. contemporary hardware
The honest findingit was memory-bound too
The lasting exportbfloat16
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
TPU vs. GPU — one does the only operation that matters and does it beautifully; the other is worse per watt and has eighteen years of software nobody can rebuild.