Support Vector Machine
Find the boundary with the widest possible gap — the method that ruled machine learning before deep learning, and still wins when data is scarce.
When not to use it
- On large datasets. The kernel matrix is n×n. Tens of thousands of rows and you're in trouble.
- On raw perceptual data. SVMs classify features; they don't learn them. That's what you lost to CNNs.
- When you need calibrated probabilities. SVMs output distances, not probabilities. Platt scaling bolts one on and it's an approximation.
- Without scaling your features. It computes distances. Unscaled features break it, and this is the most common failure.
Reach for something else instead
- Logistic regression — for wide sparse data, comparable and gives real probabilities.
- Gradient boosting — better on medium tabular data.
- Random forest — more forgiving, no scaling needed.
- Neural networks — when you have enough data and need learned representations.
Sources & further reading
- Cortes & Vapnik (1995), Support-Vector Networks — the paper, and unusually readable.
- Boser, Guyon & Vapnik (1992), A Training Algorithm for Optimal Margin Classifiers — where the kernel trick enters.
- Vapnik (1995), The Nature of Statistical Learning Theory — the theory the method came from; the margin as capacity control.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Not scaling. The single most common SVM failure, and it looks like the method not working.
- Using RBF by default on text. Linear is usually better on wide sparse data and much faster.
- Tuning C without tuning gamma. They interact strongly; grid them together.
- Expecting probabilities from
decision_function. It's a distance to the boundary, not a probability. - Reaching for one on 500,000 rows. It's the wrong tool and it will tell you slowly.
At a glance
FieldMachine Learning
Ideathe boundary with the widest margin
Best atsmall, wide datasets
Main knobsC and gamma
Requiresfeature scaling
Scalesbadly, n×n kernel matrix
Nice propertyconvex, so one optimum
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
SVM vs. logistic regression — both draw a boundary on wide data; the SVM maximises the margin and gives no probabilities, logistic regression gives probabilities and no margin guarantee.