Gradient Boosting
Trees built in sequence, each fixing the last one's mistakes — the most accurate thing on tabular data, and the easiest to overfit.
When not to use it
- Without a validation set and early stopping. Boosting reduces bias indefinitely. Nothing in the algorithm stops it fitting your noise.
- When a random forest is close enough. A few percent for a day of tuning and a fragile model isn't always the trade you want.
- On images, text, or audio. No representation learning. Wrong tool.
- When you need to explain the decision. Hundreds of sequential corrections is not an explanation.
- On very small, noisy datasets — especially with LightGBM's leaf-wise growth, which overfits fast there.
Reach for something else instead
- Random forest — more forgiving, nearly as good, no tuning.
- Regularised regression — when interpretation matters more than the last few percent.
- Neural networks — for perceptual data or where representations must be learned.
- Boosting on neural features — the hybrid that quietly wins in a lot of production systems.
Sources & further reading
- Friedman (2001), Greedy Function Approximation: A Gradient Boosting Machine — the paper that framed boosting as gradient descent in function space.
- Chen & Guestrin (2016), XGBoost: A Scalable Tree Boosting System — second-order approximation plus the engineering that made it dominate.
- Grinsztajn et al. (2022), Why do tree-based models still outperform deep learning on tabular data? — the structural case, and the one to cite when someone proposes a transformer for a spreadsheet.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- No early stopping. The single most common way to ship an overfit boosted model.
- High learning rate to "save time," then wondering why it's unstable. Low rate plus more trees is the recipe.
- Deep trees. This isn't a forest — 3 to 8 is the range, and going deeper overfits quickly.
- Choosing LightGBM for a small dataset because it's fast. Leaf-wise growth overfits small data.
- Tuning against your test set. Boosting has enough knobs that you'll succeed, and the number will be a fiction.
At a glance
FieldMachine Learning
Ideasequential trees fitting the previous errors
Reducesbias (which is why it overfits)
Essentialearly stopping
Depth3–8, shallow
Learning rate0.01–0.1
Statusthe thing to beat on tabular data
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Gradient boosting vs. random forest — sequential correction vs. parallel averaging. Boosting is sharper and will overfit; the forest is duller and won't.