CUDA
NVIDIA's platform for programming GPUs — and the actual reason NVIDIA has no competition, which is not the silicon.
Reviewed July 14, 2026Stable
When not to use it
- When PyTorch already has the operation. cuDNN and cuBLAS are hand-tuned by people who do this full time.
- Optimising arithmetic. It's memory movement. It's essentially always memory movement.
- Raw CUDA, when Triton would do. Close to the same performance, far less of your life.
- Assuming your kernel is faster than the library's. It isn't. Measure before you believe yourself.
Reach for something else instead
- Triton — Python, compiles down, handles tiling. Where custom kernels are written now.
- torch.compile — fusion without writing kernels. Try this first.
- ROCm — AMD's stack. Improved, and the ecosystem gap is the problem, not the silicon.
- XLA / TPU — a different bet entirely, and Google's.
Further reading
- Nickolls et al. (2008), Scalable Parallel Programming with CUDA — the model, from the people who built it.
- Dao et al. (2022), FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — attention was memory-bound; the kernel that proved it.
- Tillet, Kung & Cox (2019), Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations — most of the performance, a fraction of the pain.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Believing NVIDIA's moat is the silicon. It's eighteen years of libraries and everyone knowing them.
- Optimising FLOPs. FlashAttention is the proof: same maths, several times faster, purely from memory movement.
- Ignoring coalescing. Get it wrong and every other optimisation is noise.
- Writing raw CUDA in 2026 without trying Triton.
At a glance
FieldTools & Ecosystem
Released2007
The moatnot the chips; eighteen years of libraries and a generation who know them
The lessonexpert vs. naive GPU code is 10×, and it's memory movement
FlashAttentionsame maths, exact output, several times faster, purely from knowing the hierarchy
DifficultyAdvanced
Often compared with
CUDA vs. the alternatives — the silicon gap is small and closing; the ecosystem gap is eighteen years of compounding that nobody can shortcut.
Where this sits
A destination. 1 concept lead here, and nothing in the corpus depends on it.
1Levelsteps in
1Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstGPU
Computed from the prerequisite graph, not assigned. How this works