Linear Regression
The oldest and most useful model in machine learning — fit a straight line through your data — and the one every other model is secretly measured against.
Reviewed July 16, 2026Stable
When not to use it
- When the relationship is clearly non-linear and you can't fix it with feature transforms — a straight line will fit badly and mislead.
- When interpretability doesn't matter and accuracy is everything — a gradient-boosted model or network will usually win.
- With heavy outliers or heteroscedastic errors, unmodified — the least-squares fit gets dragged and the inferences break.
Reach for something else instead
- Logistic regression for predicting a category rather than a number.
- Gradient boosting when you want accuracy on tabular data and will trade away interpretability.
- Generalized additive models when you need interpretability but the relationships are curved.
Linear regression fits the line that minimises squared error.
The line is chosen to make the total squared vertical distance from the points as small as possible — the grey segments are those errors ('residuals'). Squaring punishes big misses hardest, which is why a single outlier can tilt the whole line. The fitted slope is the payoff: it tells you, in real units, how much the output changes per unit of input.
Sources & further reading
- Gauss / Legendre (c. 1805–1809) — the method of least squares, one of the oldest results in statistics.
- Hastie, Tibshirani & Friedman, The Elements of Statistical Learning — the standard treatment of linear methods and their regularised variants.
- James et al., An Introduction to Statistical Learning — the accessible version, with linear regression as the foundational chapter.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming a good fit means a causal relationship — regression finds association, not cause.
- Skipping the residual plot, so a non-linear relationship gets fit with a line and nobody notices.
- Trusting the p-values when the model's assumptions are violated.
At a glance
FieldMachine Learning
Predictsa continuous number
Fit byleast squares (closed form)
Superpowerinterpretable coefficients
DifficultyBeginner