CUDA
NVIDIA's platform for programming GPUs — and the actual reason NVIDIA has no competition, which is not the silicon.
When not to use it
- When PyTorch already has the operation. cuDNN and cuBLAS are hand-tuned by people who do this full time.
- Optimising arithmetic. It's memory movement. It's essentially always memory movement.
- Raw CUDA, when Triton would do. Close to the same performance, far less of your life.
- Assuming your kernel is faster than the library's. It isn't. Measure before you believe yourself.
Reach for something else instead
- Triton — Python, compiles down, handles tiling. Where custom kernels are written now.
- torch.compile — fusion without writing kernels. Try this first.
- ROCm — AMD's stack. Improved, and the ecosystem gap is the problem, not the silicon.
- XLA / TPU — a different bet entirely, and Google's.
Sources & further reading
- Nickolls et al. (2008), Scalable Parallel Programming with CUDA — the model, from the people who built it.
- Dao et al. (2022), FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — attention was memory-bound; the kernel that proved it.
- Tillet, Kung & Cox (2019), Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations — most of the performance, a fraction of the pain.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Believing NVIDIA's moat is the silicon. It's eighteen years of libraries and everyone knowing them.
- Optimising FLOPs. FlashAttention is the proof: same maths, several times faster, purely from memory movement.
- Ignoring coalescing. Get it wrong and every other optimisation is noise.
- Writing raw CUDA in 2026 without trying Triton.
At a glance
FieldTools & Ecosystem
Released2007
The moatnot the chips; eighteen years of libraries and a generation who know them
The lessonexpert vs. naive GPU code is 10×, and it's memory movement
FlashAttentionsame maths, exact output, several times faster, purely from knowing the hierarchy
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
CUDA vs. the alternatives — the silicon gap is small and closing; the ecosystem gap is eighteen years of compounding that nobody can shortcut.