Hyperparameter
A setting you choose rather than learn — and most of the effort spent tuning them goes into the ones that don't matter.
When not to use it
- (Tuning, that is.)*
- On parameters that don't matter. Betas, epsilon, activation choice. You're spending compute on noise.
- Before establishing a baseline. Tune after you know what the default scores, or you can't tell if it helped.
- Grid search, ever. Random search dominates it for the same budget.
- Against your test set. You will find a good configuration and the number will be fiction.
Reach for something else instead
- Better defaults — AdamW at 1e-3, cosine schedule, warmup. Often within a few percent of anything you'd find.
- μP — tune small, transfer large. What you do when you can't search.
- Hyperband — kill the losers early. Best value per unit compute.
- Removing the hyperparameter — adaptive methods that don't need it. Worth more than searching it.
Sources & further reading
- Bergstra & Bengio (2012), Random Search for Hyper-Parameter Optimization — random beats grid, and the geometric reason why.
- Li et al. (2017), Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization — kill bad configurations early; usually the best value.
- Cawley & Talbot (2010), On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation — why selecting and estimating on the same data inflates your score.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Grid search. Random search finds better configurations with the same budget, provably.
- Tuning everything equally. A few parameters dominate; the rest are decoration.
- Reporting the best cross-validated score as an unbiased estimate. You selected on it.
- Tuning before establishing a baseline, so you can't attribute the improvement.
- Treating tuning as rigour. It's what you do because the theory can't tell you.
At a glance
FieldMachine Learning
Definitionset by you, not learned from data
Key factrandom search beats grid search (Bergstra & Bengio, 2012)
What dominateslearning rate, by a distance
Best value methodHyperband
The real advanceremoving hyperparameters, not searching them
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Grid vs. random search — one tests every combination and wastes its budget on parameters that don't matter; the other explores the ones that do. Random wins for the same compute.