Field
Deep Learning
Networks with many layers, and the machinery that makes them learn.
Deep learning is the reason AI stopped being a research curiosity and started working. The core idea is old — stack simple units into layers, adjust their weights until the output is right — and the algorithm that adjusts them, backpropagation, dates to 1986.
What changed was scale. Large labelled datasets arrived, GPUs made the arithmetic cheap, and a thirty-year-old idea suddenly beat everything else. That's the pattern worth internalising: the bottleneck was rarely ideas.
This field covers the machinery. How networks learn (backpropagation, gradient descent, loss functions), the architectures that dominate (transformers, CNNs), the mechanism underneath modern AI (attention), and the shortcut nearly everyone takes (transfer learning).
Start with Neural Network if you want the ground floor, or Transformer if you want to understand what's actually running inside the tools you use.
37 concepts in this field
Transformer
The neural-network architecture behind almost every modern AI model — built around attention, which lets it weigh every word against every other, all at once.
Embeddings
Turning words (or images, or anything) into lists of numbers, arranged so that similar meanings end up close together.
Attention
The mechanism that lets an AI decide which other words matter when interpreting each word — the core idea behind transformers.
Neural Network
A system of simple connected units that learns patterns from examples — the foundation underneath deep learning and modern AI.
Backpropagation
The algorithm that works out which weights caused a mistake and by how much — the reason neural networks can learn at all.
Gradient Descent
Walking downhill on the error surface, one small step at a time — how a model's weights actually get updated.
Loss Function
The number that says how wrong the model is — and therefore the definition of what it's trying to become.
CNN (Convolutional Neural Network)
A network that slides small filters across an image to find local patterns — the architecture that made computer vision work.
Transfer Learning
Starting from a model that already learned something general, instead of from random numbers — why small teams can build real AI.
Distillation
Training a small model to imitate a large one — which works better than training the small model directly, for reasons that are still argued about.
Positional Encoding
How a transformer knows what order the words came in — a patch for the architecture's blindness to sequence, and the thing that decides how far context can stretch.
Autoencoder
A network trained to copy its input through a bottleneck — which forces it to learn what matters, and is the ancestor of most representation learning.
Activation Function
The small non-linear function after each layer — without it a hundred-layer network collapses into a single line.
Learning Rate
How big a step to take when the model updates — the single most important number in training, and the one most people leave at the default.
Optimizer
The algorithm that decides how to apply the gradient — where Adam is the default, AdamW is what you should actually use, and SGD still wins sometimes.
Batch Size
How many examples the model sees before each update — a systems constraint that everyone treats as a hyperparameter.
Dropout
Randomly switching off neurons during training — the technique that defined an era of deep learning and has quietly disappeared from modern architectures.
Batch Normalization
Renormalising activations at every layer — one of deep learning's most important techniques, and its original explanation turned out to be wrong.
Vanishing Gradient
The signal dying on its way back through a deep network — the problem that kept deep learning impossible for twenty years.
RNN (Recurrent Neural Network)
A network that reads a sequence one step at a time, carrying a memory forward — the obvious way to handle language, and the reason it took so long to work.
LSTM
An RNN with gates that decide what to remember and what to forget — the fix that made sequence learning work, and it held for twenty years.
ResNet
Add a shortcut around every couple of layers, and suddenly a hundred-layer network trains — one line of arithmetic that unlocked depth.
State Space Model
Recurrence rebuilt with control theory — constant memory, linear cost, and the most credible challenger the transformer has.
Word2Vec
The 2013 result that words could be numbers with meaningful geometry — the origin of embeddings, and its most famous demonstration was partly a trick.
Self-Supervised Learning
Learning from unlabelled data by inventing the labels from the data itself — the idea that made every modern model possible.
Perceptron
The first trainable neural network, from 1958 — and the story of how a book killed it is the most repeated wrong story in AI.
Grokking
A model that memorises, plateaus at chance on unseen data for a very long time, then abruptly generalises — and the sudden part turns out not to be sudden.
Catastrophic Forgetting
A network learning something new and losing what it already knew — a problem identified in 1989 that fine-tuning made everybody's problem again.
FlashAttention
An attention implementation that computes exactly the same answer far faster by moving less memory — and it does not make attention subquadratic.
Sigmoid Function
The S-shaped curve that squashes any number into the range 0 to 1 — the function that turns a raw score into a probability, and the historical workhorse of neural networks.
GRU
A streamlined LSTM — a recurrent network that remembers across sequences with fewer moving parts, trading a little capacity for speed and simplicity.
Self-Attention
The specific form of attention where every element of a sequence attends to every other element in the same sequence — the operation at the heart of the transformer.
Layer Normalization
Rescaling each example's activations so training stays stable — the normalization transformers actually use, and the reason batch normalization did not fit them.
Softmax
Turning a list of arbitrary numbers into a probability distribution — the operation at the end of every classifier and inside every attention head.
Encoder-Decoder
Read the whole input into a representation, then generate the output from it — the architecture that made translation work and that attention was invented to fix.
Cross-Attention
Attention where the queries come from one sequence and the keys and values from another — the mechanism that lets a decoder read an encoder, and the one that makes multimodal models possible.
Residual Connection
Adding a layer's input to its output so gradients have an unobstructed path backwards — the single change that made networks deeper than about twenty layers trainable at all.
Read more on the blog
- Where the attention mechanism actually came fromAttention was a fix for a specific engineering failure in 2014, three years before the paper that made it famous. It was not designed as a theory of cognition, and the name was applied afterwards by analogy.
- Your RAG system isn't hallucinating. It never found the answer.When a RAG system gives a bad answer, almost everyone blames the model. Usually the right passage was never retrieved, and that changes everything about how you fix it.
- I spent a month writing down what every AI concept depends on256 concepts, 350 prerequisite links, one rule. I wrote down my predictions first so I could not quietly retrofit the results. Two of them were wrong, and those are the interesting part.
- Who invented deep learning, and why it took so longBackpropagation was invented at least four times before it stuck. The ideas behind deep learning were mostly in place by 1990. What was missing was not insight.