Regression
Predicting a number rather than a category — the oldest tool in the box, and still the right answer more often than anyone admits.
When not to use it
- When the relationship is genuinely non-linear and you can't feature-engineer around it. Forcing a line through a curve gives you a model that's wrong in a specific, patterned way.
- When you have many interacting features. Trees find interactions automatically; regression needs you to specify each one.
- When interpretation doesn't matter and accuracy does. If nobody will ever ask why, you're giving up performance for a property you're not using.
- On heavily multicollinear features, if you intend to explain the coefficients. The prediction survives; the explanation doesn't.
Reach for something else instead
- Gradient boosting — better accuracy on tabular data, no explanation.
- Generalised additive models — non-linear per feature, still interpretable. The underused middle ground.
- Decision trees — interpretable and non-linear, at some accuracy cost.
- Quantile regression — when you need a range rather than a point, or your data has tails.
Sources & further reading
- Breiman (2001), Statistical Modeling: The Two Cultures — the essay that named the split between explaining and predicting. Read it once.
- Hastie, Tibshirani & Friedman (2009), The Elements of Statistical Learning — the reference; the regression chapters are still the clearest treatment.
- Tibshirani (1996), Regression Shrinkage and Selection via the Lasso — L1 regularisation, and getting feature selection for free.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Not plotting residuals. A patterned residual plot is the model telling you it's missing something, and it's ignored constantly.
- Explaining coefficients from a multicollinear model. They're unstable and the story you tell will be wrong.
- Reporting p-values from a model whose assumptions are violated. The number appears; the meaning doesn't.
- Skipping linear regression as a baseline. You need to know what the simple thing scored before you claim the complex thing helped.
- Confusing logistic regression with regression. It's classification wearing the name.
At a glance
FieldMachine Learning
Predictsa number
Agetwo centuries
Main assetthe answer is a sentence
Main riskmulticollinearity ruining the explanation
Alwaysplot the residuals
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Linear vs. logistic regression — one predicts a number, the other a probability of a class. Only the first is regression despite the shared name.