Artifipedia
GlossaryMapBlog
Home/Deep Learning/Dropout
Deep Learning

Dropout

Randomly switching off neurons during training — the technique that defined an era of deep learning and has quietly disappeared from modern architectures.

Reading level: Curious
Pick your depth ↓

When not to use it

  • In modern transformers, by default. They use very little or none, and large models often train with zero.
  • On large datasets. It addresses overfitting, and overfitting isn't your constraint.
  • Alongside batch normalization, carelessly. The variance shift between training and inference can make the pair worse than either.
  • On convolutional layers, naively. Spatial correlation means dropping individual activations achieves little.

Reach for something else instead

  • More data — the thing dropout was substituting for.
  • Weight decay — covers much of the same ground and interacts better with modern architectures.
  • Data augmentation — usually more effective on vision.
  • Early stopping — free.
  • Layer normalization — what modern architectures use instead.

Sources & further reading

  • Srivastava et al. (2014), Dropout: A Simple Way to Prevent Neural Networks from Overfitting — the paper, and the ensemble story.
  • Li et al. (2019), Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift — why the two together can hurt.
  • Gal & Ghahramani (2016), Dropout as a Bayesian Approximation — MC Dropout; the reinterpretation, and it's contested.

Primary sources, listed so you can check the claims on this page rather than take them on trust.

Where people go wrong

  • Forgetting model.eval(). Dropout stays on at inference and your model is randomly, non-deterministically worse.
  • Using 0.5 in a transformer because it was the classic default. It's for fully-connected layers on small data.
  • Stacking it with batch normalization without knowing about the variance shift.
  • Repeating the ensemble explanation as established. It's exact only for linear models and it survives because it's memorable.

At a glance

FieldDeep Learning
What it doesrandomly zeroes units during training
Classic rate0.5 for FC layers; 0-0.1 in modern transformers
Statuslargely absent from current architectures
Classic bugforgetting model.eval()
Mechanismcontested; the ensemble story is loose
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4

Often compared with

Dropout vs. weight decay — one deletes units randomly, the other shrinks them smoothly. Modern architectures kept the second.

Related concepts

Machine Learning

Regularization

Anything that stops a model fitting the training data too well — a collection of tricks, held together by a story that modern deep learning broke.

Machine Learning

Overfitting

When a model memorizes its training data instead of learning the general pattern — so it looks great in training but fails on new data.

Deep Learning

Neural Network

A system of simple connected units that learns patterns from examples — the foundation underneath deep learning and modern AI.

Deep Learning

Batch Normalization

Renormalising activations at every layer — one of deep learning's most important techniques, and its original explanation turned out to be wrong.

Foundations

Deep Learning

Machine learning using neural networks with many layers — the approach behind nearly every recent AI breakthrough.

Deep Learning

Transformer

The neural-network architecture behind almost every modern AI model — built around attention, which lets it weigh every word against every other, all at once.