Artifipedia
GlossaryMapStudyBlogUniversitiesMy Library
Home/Deep Learning/Dropout
Deep Learning

Dropout

Randomly switching off neurons during training — the technique that defined an era of deep learning and has quietly disappeared from modern architectures.

Reviewed July 12, 2026Contested
Reading level: Curious
Pick your depth ↓

When not to use it

  • In modern transformers, by default. They use very little or none, and large models often train with zero.
  • On large datasets. It addresses overfitting, and overfitting isn't your constraint.
  • Alongside batch normalization, carelessly. The variance shift between training and inference can make the pair worse than either.
  • On convolutional layers, naively. Spatial correlation means dropping individual activations achieves little.

Reach for something else instead

  • More data — the thing dropout was substituting for.
  • Weight decay — covers much of the same ground and interacts better with modern architectures.
  • Data augmentation — usually more effective on vision.
  • Early stopping — free.
  • Layer normalization — what modern architectures use instead.

Dropout randomly switches off neurons during training.

dashed = dropped this step

At each training step, dropout randomly deactivates a fraction of neurons, so the network can't lean on any single one and must learn redundant, robust features. It's like training a slightly different smaller network each time and averaging them — a cheap, powerful defence against overfitting. At test time all neurons are back on.

Further reading

  • Srivastava et al. (2014), Dropout: A Simple Way to Prevent Neural Networks from Overfitting — the paper, and the ensemble story.
  • Li et al. (2019), Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift — why the two together can hurt.
  • Gal & Ghahramani (2016), Dropout as a Bayesian Approximation — MC Dropout; the reinterpretation, and it's contested.

Primary sources, listed so you can check the claims on this page rather than take them on trust.

Where people go wrong

  • Forgetting model.eval(). Dropout stays on at inference and your model is randomly, non-deterministically worse.
  • Using 0.5 in a transformer because it was the classic default. It's for fully-connected layers on small data.
  • Stacking it with batch normalization without knowing about the variance shift.
  • Repeating the ensemble explanation as established. It's exact only for linear models and it survives because it's memorable.

At a glance

FieldDeep Learning
What it doesrandomly zeroes units during training
Classic rate0.5 for FC layers; 0-0.1 in modern transformers
Statuslargely absent from current architectures
Classic bugforgetting model.eval()
Mechanismcontested; the ensemble story is loose
DifficultyBeginner
Flashcards for this concept · study, save or share them →
Question
—
Answer
—
1 / 4

Often compared with

Dropout vs. weight decay — one deletes units randomly, the other shrinks them smoothly. Modern architectures kept the second.

Where this sits

A destination. 8 concepts lead here, and nothing in the corpus depends on it.

6Levelsteps in
8Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstNeural NetworkRegularization
LEARN FIRST Neural Network Regularization Dropout
Dropout sits after Neural Network and Regularization, and nothing further depends on it.

Computed from the prerequisite graph, not assigned. How this works

Related concepts

Machine Learning

Regularization

Anything that stops a model fitting the training data too well — a collection of tricks, held together by a story that modern deep learning broke.

Machine Learning

Overfitting

When a model memorizes its training data instead of learning the general pattern — so it looks great in training but fails on new data.

Deep Learning

Neural Network

A system of simple connected units that learns patterns from examples — the foundation underneath deep learning and modern AI.

Deep Learning

Batch Normalization

Renormalising activations at every layer — one of deep learning's most important techniques, and its original explanation turned out to be wrong.

Foundations

Deep Learning

Machine learning using neural networks with many layers — the approach behind nearly every recent AI breakthrough.

Deep Learning

Transformer

The neural-network architecture behind almost every modern AI model — built around attention, which lets it weigh every word against every other, all at once.