Neural Network
A system of simple connected units that learns patterns from examples — the foundation underneath deep learning and modern AI.
When not to use it
- On tabular data. Gradient-boosted trees still beat neural networks on most spreadsheet-shaped problems, train in seconds, and explain themselves.
- With small datasets. A few hundred rows and a neural network is a recipe for memorising noise. Simpler models generalise better when data is scarce.
- When you must justify each decision. "The weights say so" doesn't survive a regulator, a clinician, or a loan applicant.
Reach for something else instead
- Gradient boosting (XGBoost, LightGBM) — the honest default for tabular prediction.
- Linear and logistic regression when interpretability is the requirement, not an afterthought.
- Classical algorithms — sometimes the task is a sort, a join, or a rule, and no learning is needed at all.
Sources & further reading
- Rumelhart, Hinton & Williams (1986), Learning representations by back-propagating errors — the algorithm everything still runs on.
- Grinsztajn, Oyallon & Varoquaux (2022), Why do tree-based models still outperform deep learning on tabular data? — the paper to cite when someone reaches for a neural net on a spreadsheet.
- LeCun, Bengio & Hinton (2015), Deep Learning (Nature) — the field's own summary of why depth mattered.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Adding layers to fix a data problem. More capacity memorises faster; it doesn't understand better.
- Skipping the simple baseline, so nobody knows whether the network is actually earning its complexity.
- Confusing training loss going down with the model getting good. That's the definition of overfitting, watched in real time.
At a glance
FieldDeep Learning
Core idealearn patterns from examples
Learns viabackpropagation + gradient descent
Needsdata and compute
DifficultyBeginner → Intermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4