Field
Deep Learning
Networks with many layers, and the machinery that makes them learn.
Deep learning is the reason AI stopped being a research curiosity and started working. The core idea is old — stack simple units into layers, adjust their weights until the output is right — and the algorithm that adjusts them, backpropagation, dates to 1986.
What changed was scale. Large labelled datasets arrived, GPUs made the arithmetic cheap, and a thirty-year-old idea suddenly beat everything else. That's the pattern worth internalising: the bottleneck was rarely ideas.
This field covers the machinery. How networks learn (backpropagation, gradient descent, loss functions), the architectures that dominate (transformers, CNNs), the mechanism underneath modern AI (attention), and the shortcut nearly everyone takes (transfer learning).
Start with Neural Network if you want the ground floor, or Transformer if you want to understand what's actually running inside the tools you use.
26 concepts in this field
Transformer
The neural-network architecture behind almost every modern AI model — built around attention, which lets it weigh every word against every other, all at once.
Embeddings
Turning words (or images, or anything) into lists of numbers, arranged so that similar meanings end up close together.
Attention
The mechanism that lets an AI decide which other words matter when interpreting each word — the core idea behind transformers.
Neural Network
A system of simple connected units that learns patterns from examples — the foundation underneath deep learning and modern AI.
Backpropagation
The algorithm that works out which weights caused a mistake and by how much — the reason neural networks can learn at all.
Gradient Descent
Walking downhill on the error surface, one small step at a time — how a model's weights actually get updated.
Loss Function
The number that says how wrong the model is — and therefore the definition of what it's trying to become.
CNN (Convolutional Neural Network)
A network that slides small filters across an image to find local patterns — the architecture that made computer vision work.
Transfer Learning
Starting from a model that already learned something general, instead of from random numbers — why small teams can build real AI.
Distillation
Training a small model to imitate a large one — which works better than training the small model directly, for reasons that are still argued about.
Positional Encoding
How a transformer knows what order the words came in — a patch for the architecture's blindness to sequence, and the thing that decides how far context can stretch.
Autoencoder
A network trained to copy its input through a bottleneck — which forces it to learn what matters, and is the ancestor of most representation learning.
Activation Function
The small non-linear function after each layer — without it a hundred-layer network collapses into a single line.
Learning Rate
How big a step to take when the model updates — the single most important number in training, and the one most people leave at the default.
Optimizer
The algorithm that decides how to apply the gradient — where Adam is the default, AdamW is what you should actually use, and SGD still wins sometimes.
Batch Size
How many examples the model sees before each update — a systems constraint that everyone treats as a hyperparameter.
Dropout
Randomly switching off neurons during training — the technique that defined an era of deep learning and has quietly disappeared from modern architectures.
Batch Normalization
Renormalising activations at every layer — one of deep learning's most important techniques, and its original explanation turned out to be wrong.
Vanishing Gradient
The signal dying on its way back through a deep network — the problem that kept deep learning impossible for twenty years.
RNN (Recurrent Neural Network)
A network that reads a sequence one step at a time, carrying a memory forward — the obvious way to handle language, and the reason it took so long to work.
LSTM
An RNN with gates that decide what to remember and what to forget — the fix that made sequence learning work, and it held for twenty years.
ResNet
Add a shortcut around every couple of layers, and suddenly a hundred-layer network trains — one line of arithmetic that unlocked depth.
State Space Model
Recurrence rebuilt with control theory — constant memory, linear cost, and the most credible challenger the transformer has.
Word2Vec
The 2013 result that words could be numbers with meaningful geometry — the origin of embeddings, and its most famous demonstration was partly a trick.
Self-Supervised Learning
Learning from unlabelled data by inventing the labels from the data itself — the idea that made every modern model possible.
Perceptron
The first trainable neural network, from 1958 — and the story of how a book killed it is the most repeated wrong story in AI.