Reference
The glossary.
Every AI term you might meet and not know, defined in a sentence. 167 of them link through to a full entry, explained at five depths.
180 terms
A
Ablation
Removing one part of a system to see how much it mattered. The standard way papers show a component earns its place.
The share of predictions that were right. Misleading whenever classes are imbalanced: predicting "healthy" for everyone scores 99% on a rare disease.
The small non-linear step applied after each layer. Without it, stacking layers would collapse into a single linear operation and depth would buy nothing.
The default optimiser. Adapts the step size per parameter, which is why it works without much tuning.
Adversarial example
An input nudged, often imperceptibly, to make a model fail. Evidence that models rely on patterns humans don't see.
A system that decides what to do next rather than answering once — usually a model in a loop with tools.
How an agent remembers across sessions, since the model itself forgets everything. Retrieval and storage, not learning.
Hypothetical AI with broad human-level capability. Undefined enough that people argue about whether it has arrived.
The field of making machines do things that seem to require intelligence. The definition moves every time machines succeed.
The problem of getting AI systems to actually do what people intend.
Finding close enough matches fast instead of exact matches slowly. What every vector database runs on.
The interface you send a request to and get a model's answer back from. How nearly everyone uses AI.
The mechanism letting a model weigh every token against every other. The transformer's core idea.
AUC / ROC
A measure of how well a classifier separates classes across all thresholds. Useful when you haven't picked a threshold yet.
Generating one token at a time, each conditioned on everything before it. Why models write at a steady pace rather than all at once.
B
The algorithm that works out which weights caused a mistake, by tracing error backwards through the network.
A model after pretraining but before instruction tuning. Completes text rather than answering questions.
Baseline
The simple thing you must beat before claiming anything. Often a rule, a linear model, or "predict the most common answer."
The group of examples processed together in one step. Bigger batches give smoother gradients and cost more memory.
How many examples per training step. Interacts with learning rate; copying one without the other means nothing.
Keeping several candidate continuations alive and picking the best complete one. Standard in translation, mostly abandoned for chat.
A standard test set used to compare models. Increasingly unreliable, since the test may be inside the training data.
An early influential language model that reads text in both directions. Still used for classification and search, not generation.
A system producing unfair outcomes across groups of people.
Systematic error — the model is consistently wrong in one direction, regardless of data.
The compression trick used to split text into tokens. Started as a 1994 compression algorithm.
Broadcasting
Automatically stretching arrays of different shapes to work together. A NumPy convention every ML framework inherited.
C
When training on something new destroys what a model already knew. The main risk in fine-tuning.
Getting a model to reason step by step before answering. Improves results, and the stated reasoning isn't always the real one.
Checkpoint
A saved snapshot of a model's weights mid-training. Also what you pin to when you need reproducibility.
Splitting documents into pieces before indexing them. The most common invisible cause of bad RAG.
Predicting which category something belongs to.
A model that puts images and text in the same embedding space, so you can compare them. Underpins most image search and generation.
Grouping similar things without labels. Always returns groups, whether or not any exist.
A network that slides small filters across an image to find local patterns. Made computer vision work.
How much text a model can consider at once. Bigger isn't the same as better-used.
Training by pulling matching pairs together and pushing non-matching pairs apart. How CLIP and most embedding models learn.
When training stops improving. Not the same as being finished, and often a plateau rather than an answer.
Corpus
The body of text a model was trained on, or that you search over.
Measuring how close two embeddings point in the same direction. The standard way to compare meanings numerically.
The standard loss for classification. A differentiable stand-in for accuracy, which has no useful gradient.
Rotating which slice of data is held out, to get a more reliable estimate on small datasets.
NVIDIA's software layer for running computation on GPUs. Its maturity, not the silicon, is why alternatives struggle.
D
Making more training examples by flipping, cropping, or perturbing existing ones. Often beats a fancier architecture.
When the data in production stops resembling the data you trained on. Why models decay after launch.
The half of a model that generates output. Modern LLMs are decoder-only.
Machine learning with many-layered neural networks.
Synthetic media convincingly depicting a real person doing something they didn't. Descended from GANs.
Generates images by starting from noise and removing it step by step. Replaced GANs for most image work.
Squeezing many features into fewer while keeping the structure. PCA, UMAP, t-SNE.
Training a small model to imitate a big one. Better quality at size than heavy quantization, at the cost of a training run.
Distribution
The shape of your data — what's typical, what's rare. Almost every model failure is a distribution mismatch.
Direct preference optimisation. Trains on preferences without a separate reward model or an RL loop.
Randomly switching off units during training to prevent memorisation.
E
Halting training when validation performance stops improving, so the model doesn't start memorising.
A list of numbers representing meaning, so similar things sit close together.
The half of a model that reads and represents input.
Ensemble
Combining several models and averaging them. Usually more accurate, always more expensive.
One full pass through the training data.
How you find out whether it works. The step everyone skips and then argues about vibes.
Getting a reason out of a model. Most methods explain the explanation, not the decision.
When the training signal compounds until numbers overflow. Fixed by clipping.
F
F1 score
The balance of precision and recall in one number. What you usually care about when accuracy lies.
One input column or measurement the model sees.
Turning raw data into things a model can use. Still where most accuracy comes from outside deep learning.
Putting a handful of examples in the prompt. Free, instant, and skipping it is the most common expensive mistake.
A score for image generation quality. Correlates with human judgement loosely enough to be worth distrusting.
Continuing a model's training on your own examples to change its behaviour.
A reorganisation of attention's memory access that made long contexts affordable. Same maths, less waiting.
A large model pretrained on broad data, meant to be adapted to many tasks.
Giving a model a described set of functions it can request. The mechanism under tool use.
G
Two networks trained against each other, one faking and one detecting. Largely replaced by diffusion.
Working on data you've never seen. The entire point; everything else is bookkeeping.
The chip that made deep learning possible — thousands of cores doing the same maths at once.
The direction and steepness of the error surface at your current position. What tells the model which way to move.
Capping the size of updates so one bad batch can't destroy a training run.
Walking downhill on the error surface, one step at a time.
The correct answer you're comparing against. Only as good as whoever labelled it.
The checks around a model deciding what it can receive, say, and do.
H
When a model produces something fluent, confident, and untrue. Frequently misdiagnosed — often the retrieval failed.
The graph index most vector databases actually run. Fast approximate search over embeddings.
Combining keyword and semantic search. Close to free and reliably better than either alone.
A setting you choose rather than one the model learns — learning rate, batch size, layer count.
I
Predicting what an image is. The task that started the deep learning era in 2012.
A model adapting its behaviour from examples in the prompt, with no training.
Using a trained model to get an answer. Where the money goes at scale.
Labelling every pixel and separating individual objects, so you can count them.
Training a base model to follow instructions rather than merely continue text.
The word underneath "artificial intelligence," used constantly and defined by nobody. Over seventy published definitions and counting.
How much a predicted region overlaps the true one. The standard metric for detection and segmentation.
J
Getting a model to do what it was trained to refuse. Persists because refusals are tendencies, not rules.
K
The default clustering algorithm. Assumes round, similar-sized groups, and finds them whether or not they're there.
The date after which a model knows nothing. Why it's confident and wrong about recent events.
Stored attention keys and values reused while generating. Grows with context length, which is why long prompts get expensive to serve.
L
The correct answer attached to a training example.
The compressed internal representation a model works in. Where diffusion models do their denoising.
One transformation stage in a network. Depth is how many.
Rescaling activations within a layer to keep training stable. Quiet, essential, invisible.
Building a feature from information unavailable at prediction time. Produces brilliant validation scores and useless models.
How big a step to take downhill. The single most important dial in training.
A model trained on enormous text corpora to predict what comes next, and capable of far more than that suggests.
The raw scores a model outputs before they're turned into probabilities. What temperature is applied to.
Low-rank adaptation. Fine-tuning by training a small add-on rather than the whole model. The sensible default.
The number saying how wrong the model is, and therefore the definition of what it's trying to become.
M
Getting computers to learn patterns from data instead of being programmed with rules.
Mean absolute and mean squared error. MSE squares mistakes, so it obsesses over outliers.
When a model reproduces training data rather than generalising. A privacy problem as well as a quality one.
Routing each token to a few specialised sub-networks instead of the whole model. More parameters, similar cost per token.
When a generator finds a few outputs that work and produces only those. The characteristic GAN failure.
Documentation of what a model is, what it was trained on, and where it fails.
Accumulating velocity across training steps, damping oscillation.
Several agents working on one problem. Strong in demos, awkward in production; coordination costs are real.
Handling more than one kind of input — text and images, sometimes audio or video.
N
Finding names, places, and organisations in text. Standard first step in PII detection.
Layers of simple units whose connection strengths are adjusted until the output is useful.
Rescaling values to a common range so no feature dominates by magnitude alone.
Sampling only from the smallest set of tokens whose probabilities sum to p. Better text than pure likelihood.
O
Finding where things are in an image and what they are. Boxes, not outlines.
Turning pictures of text into text. Solved for clean scans, genuinely hard for everything else.
Representing a category as a vector of zeros with a single one. The naive alternative to embeddings.
A model whose weights you can download and run. Usually not open source, whatever the announcement says.
The algorithm that decides how to apply gradients. Adam unless you have a reason.
Learning the training data instead of the pattern. Perfect on what it's seen, useless on what it hasn't.
P
One learned number inside a model. Counted in billions, and a poor proxy for capability.
Principal component analysis. The classic way to reduce dimensions while keeping variance.
How surprised a model is by text. Lower is better, and it hides exactly the capabilities you care about.
Personally identifiable information. The category that turns a data question into a legal one.
How a model knows word order, since attention alone sees a set rather than a sequence.
Precision
Of the things you flagged, how many were right.
The first, enormous training run on broad data, before any task-specific work.
What you send a model.
Reusing computation for a repeated prefix. Often the single biggest cost saving available.
Writing inputs that get better outputs. Less mystical and more empirical than the name suggests.
Instructions hidden in content a model reads, which it follows as if you'd written them. Unsolved in the general case.
Removing weights that don't matter to shrink a model.
Q
Storing a model's numbers with less precision, so it fits in less memory and runs faster.
The three projections attention uses to decide what attends to what.
R
Fetching relevant documents at query time and letting the model answer from them.
Of the things you should have found, how many you did. The ceiling on any RAG system.
Attacking your own system deliberately, before someone else does it for free.
Anything that discourages memorisation — dropout, weight decay, early stopping.
Retrieving a wide net, then reordering with a sharper model and keeping the best few. The best value in a RAG pipeline.
A shortcut letting signal skip a layer. The reason very deep networks train at all.
Optimising the measure instead of the goal. Not a bug — what optimisation does.
A learned stand-in for human judgement. RLHF's weak link by construction.
Training on human preferences rather than correct answers. The step that turned text predictors into assistants.
The sequential architecture transformers replaced. Processed text one step at a time, which is why it was slow.
S
Choosing the next token from a probability distribution rather than always taking the likeliest.
The observed relationship between compute, data, model size, and performance. Corrected once already.
Attention within one sequence — every token weighing every other.
Inventing labels from the data's own structure. How foundation models are actually trained.
Searching by meaning rather than exact words. Similar is not the same as relevant.
Stochastic gradient descent. Computing the gradient on a small random batch — noisy, cheap, and the noise helps.
A feature attribution method. Better grounded than most, and its independence assumptions break on correlated data.
Turning raw scores into probabilities that sum to one. Where temperature is applied.
SOTA
State of the art. Claimed in every release, true on the chart they chose, for a few weeks.
A model agreeing with you because agreement was preferred by labellers. Learned, not accidental.
Standing instructions given before the conversation. Not a secret, and not a security boundary.
T
Methods for visualising high-dimensional data. Cluster sizes and distances on the plot are not meaningful.
The randomness dial. Low is predictable, high is varied, and it isn't a creativity setting.
Tensor
A multi-dimensional array. The data structure everything in deep learning is made of.
Data you look at once, at the end. Check it repeatedly and you no longer have one.
The chunk of text a model actually reads — usually part of a word. What you're billed for.
The thing that splits text into tokens. Explains why models miscount letters.
Letting a model call real software instead of answering from memory.
Sampling only from the k likeliest tokens.
Building the model. Expensive, memory-hungry, and something most teams should never do.
Starting from a model that already learned something general. Why small teams can ship real AI.
The architecture behind almost every modern model, built around attention.
U
The model is too simple to capture the pattern. Bad on training data and test data alike.
Removing a specific record's influence from a trained model. No reliable general method exists.
Finding structure in data nobody labelled. Hard to know whether you got it right.
V
The data you tune against, kept separate from the test set.
The training signal shrinking as it travels back, until early layers learn nothing.
A store built to find the nearest embeddings fast. Often reached for before it's needed.
A transformer applied to image patches. Beats CNNs given enough data, and only then.
W
One learned number determining how strongly one unit influences another.
Penalising large weights to discourage memorisation.
The 2013 model that made embeddings famous, and the source of king − man + woman ≈ queen.
Z
Doing a task with no examples, from instructions alone.
Terms with a full entry link to a concept page — the same idea at five depths, with sources and when not to use it. The rest are here because you'd meet them and want one sentence.