Field
Machine Learning
Learning patterns from data — the foundation everything else sits on.
Machine learning is the broader field that deep learning is part of. Most working ML has nothing to do with neural networks: it's gradient-boosted trees on tabular data, quietly outperforming anything fancier and doing so in seconds.
That's worth stating plainly, because the assumption that deep learning is always better is expensive and wrong. On spreadsheet-shaped problems it usually loses, and there's a well-cited paper saying so.
This field covers the fundamentals that don't change: supervised and unsupervised learning, clustering, feature engineering — still where most of the accuracy comes from outside deep learning — and the two ideas that cause the most real-world failure: overfitting and getting your train/test split wrong.
Start with Machine Learning, then Overfitting — it explains more failures than anything else here.
31 concepts in this field
Supervised Learning
Teaching an AI by showing it labelled examples — inputs paired with the correct answers — so it can predict answers for new inputs.
Overfitting
When a model memorizes its training data instead of learning the general pattern — so it looks great in training but fails on new data.
Unsupervised Learning
Finding structure in data nobody labelled — useful, underrated, and much harder to know if you got right.
Clustering
Grouping things that resemble each other — and the fact that the algorithm always returns groups, whether or not any exist.
Feature Engineering
Reshaping raw data into things a model can actually use — still where most of the accuracy comes from outside deep learning.
Train/Test Split
Holding back data the model never sees, so you can find out whether it learned anything or just memorised.
Precision and Recall
The two ways to be right and the two ways to be wrong — and the trade-off that accuracy hides from you.
Cross-Validation
Testing on every part of your data by rotating which part you hold back — the fix for "my score depends on which rows I happened to set aside."
Regression
Predicting a number rather than a category — the oldest tool in the box, and still the right answer more often than anyone admits.
Decision Tree
A flowchart learned from data — the most interpretable model there is, and on its own, one of the least accurate.
Random Forest
Hundreds of deliberately mediocre trees, averaged — the strongest default in machine learning, and almost impossible to misuse.
Gradient Boosting
Trees built in sequence, each fixing the last one's mistakes — the most accurate thing on tabular data, and the easiest to overfit.
Support Vector Machine
Find the boundary with the widest possible gap — the method that ruled machine learning before deep learning, and still wins when data is scarce.
K-Nearest Neighbours
Predict by looking at the most similar examples you've already seen — no training at all, and the ancestor of every vector search you use today.
Bias-Variance Tradeoff
The two ways a model can be wrong, and the classical claim that fixing one worsens the other — which modern deep learning appears to violate.
Dimensionality Reduction
Squashing many features into few — useful for compression and computation, and dangerous the moment you believe the picture.
Regularization
Anything that stops a model fitting the training data too well — a collection of tricks, held together by a story that modern deep learning broke.
Hyperparameter
A setting you choose rather than learn — and most of the effort spent tuning them goes into the ones that don't matter.
Confusion Matrix
A table of what got classified as what — the least sophisticated tool in evaluation, and the one that tells you the most.
F1 Score
The harmonic mean of precision and recall — the default single number for classification, and it encodes a decision nobody made.
ROC and AUC
A curve showing every threshold at once, summarised into one number — the most-reported classification metric, and it has a coherence problem almost nobody knows about.
Calibration
Whether a model's confidence means anything — and modern neural networks are worse at it than the ones they replaced.
Inter-annotator Agreement
How often your human labellers agree with each other — the real ceiling on your model, and the number most projects never compute.
Training Data
The examples a model learns from — where almost all of its capability and almost all of its failures come from, and the part of the work nobody wants to do.
Data Labeling
Humans deciding what each example is — the least visible and most determinative work in supervised learning, done by people the field rarely names.
Synthetic Data
Training on data a model generated — increasingly standard, genuinely useful, and carrying a failure mode with a Nature paper attached.
Data Augmentation
Making more training examples by transforming the ones you have — the most effective regularizer there is, and it encodes assumptions you should state out loud.
Class Imbalance
When one class vastly outnumbers another — and the standard advice to resample is mostly wrong.
Bayesian Inference
Updating beliefs with evidence, according to the only rule that's coherent — mathematically settled, practically expensive, and the thing modern models are bad at.
Inductive Bias
The assumptions a model makes before seeing any data — without them learning is impossible, and there's a theorem.
Generalization
Working on data you've never seen — the only thing that matters, and nobody can explain why deep learning does it.