Decision Tree
A flowchart learned from data — the most interpretable model there is, and on its own, one of the least accurate.
When not to use it
- As a production model, alone. A single tree loses to a forest or boosting on almost every dataset. Use the ensemble unless the tree itself is the deliverable.
- When the true boundary is diagonal or smooth. Axis-aligned splits approximate a diagonal with a staircase — many splits, poor fit.
- When stability matters. Retrain on slightly different data and get a structurally different tree. That's hard to explain to stakeholders who read the last one.
- On very wide, sparse data. Text, high-dimensional embeddings. Trees struggle where linear models do fine.
Reach for something else instead
- Random forest — the same trees, averaged, far more accurate and stable.
- Gradient boosting — usually the accuracy winner on tabular data.
- Rule lists / scoring systems — interpretable by construction and often competitive.
- Logistic regression — interpretable, stable, and better on wide sparse data.
Sources & further reading
- Breiman et al. (1984), Classification and Regression Trees — CART; the foundational treatment.
- Quinlan (1986), Induction of Decision Trees — ID3, the other lineage, and where entropy-based splitting comes from.
- Rudin (2019), Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead — the argument for trees over post-hoc explanation, made seriously.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Shipping an unconstrained tree. It memorised your training set and you'll find out in production.
- Agonising over Gini vs. entropy. They rarely disagree enough to matter; the depth limit does.
- Trusting default feature importance. It's biased toward high-cardinality columns — an ID field will look predictive.
- Expecting stability. Trees are high-variance by construction; that's the property ensembles exist to exploit.
At a glance
FieldMachine Learning
Structurerecursive yes/no splits
Strengthfully interpretable
Weaknesshigh variance, low accuracy alone
Key knobmax depth / min samples per leaf
Real useas an ensemble component
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Decision tree vs. random forest — the same model, one versus hundreds averaged. You trade the interpretability you came for to fix the variance you didn't want.