Support Vector Machine
Find the boundary with the widest possible gap — the method that ruled machine learning before deep learning, and still wins when data is scarce.
When not to use it
- On large datasets. The kernel matrix is n×n. Tens of thousands of rows and you're in trouble.
- On raw perceptual data. SVMs classify features; they don't learn them. That's what you lost to CNNs.
- When you need calibrated probabilities. SVMs output distances, not probabilities. Platt scaling bolts one on and it's an approximation.
- Without scaling your features. It computes distances. Unscaled features break it, and this is the most common failure.
Reach for something else instead
- Logistic regression — for wide sparse data, comparable and gives real probabilities.
- Gradient boosting — better on medium tabular data.
- Random forest — more forgiving, no scaling needed.
- Neural networks — when you have enough data and need learned representations.
An SVM finds the widest street separating two classes.
A support vector machine draws the boundary that maximises the margin — the gap between the two classes — rather than just any separating line. Only the points on the edge (the support vectors) determine it; the rest are irrelevant. That wide-margin principle gives strong generalisation, and the kernel trick lets it draw curved boundaries by working in a higher-dimensional space.
Further reading
- Cortes & Vapnik (1995), Support-Vector Networks — the paper, and unusually readable.
- Boser, Guyon & Vapnik (1992), A Training Algorithm for Optimal Margin Classifiers — where the kernel trick enters.
- Vapnik (1995), The Nature of Statistical Learning Theory — the theory the method came from; the margin as capacity control.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Not scaling. The single most common SVM failure, and it looks like the method not working.
- Using RBF by default on text. Linear is usually better on wide sparse data and much faster.
- Tuning C without tuning gamma. They interact strongly; grid them together.
- Expecting probabilities from
decision_function. It's a distance to the boundary, not a probability. - Reaching for one on 500,000 rows. It's the wrong tool and it will tell you slowly.
At a glance
Often compared with
Where this sits
A destination. 2 concepts lead here, and nothing in the corpus depends on it.
Computed from the prerequisite graph, not assigned. How this works