Feature Engineering
Reshaping raw data into things a model can actually use — still where most of the accuracy comes from outside deep learning.
When not to use it
- On images, audio, or text. Learned representations beat hand-built features there decisively. Fine-tune a pretrained model instead.
- Before you have a baseline. Engineer features to beat something, not in a vacuum — otherwise you can't tell what helped.
- When the feature can't be computed at prediction time. That's leakage, and it produces a validation score you'll believe and a model that fails.
Reach for something else instead
- Deep learning on unstructured data — the whole point of it is learning the features.
- Automated feature tools for breadth, if you accept that they generate volume and you'll still do the selection.
- Better data collection — sometimes the missing signal isn't derivable from what you have, and no transformation invents it.
Sources & further reading
- Kaufman et al. (2012), Leakage in Data Mining: Formulation, Detection, and Avoidance — the failure that explains most implausibly good results.
- Grinsztajn, Oyallon & Varoquaux (2022), Why do tree-based models still outperform deep learning on tabular data? — why this work still matters where it matters.
- Domingos (2012), A Few Useful Things to Know About Machine Learning — "feature engineering is the key," from someone with standing to say it.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Building features from data that only exists after the outcome. 99% accuracy is a symptom, not a success.
- Target encoding without out-of-fold computation, leaking the label into the feature by construction.
- Computing features differently in the notebook and in production, then debugging a model that "got worse after deployment" when it never changed.
At a glance
FieldMachine Learning
Core ideaturn raw data into signal
Matters mosttabular data
Matters leastimages, text, audio
Chief hazardleakage
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Feature engineering vs. deep learning — encoding what you know vs. learning it from data you have.