Artifipedia
GlossaryMapStudyBlogUniversitiesMy Library
Home/Deep Learning/Batch Size
Deep Learning

Batch Size

How many examples the model sees before each update — a systems constraint that everyone treats as a hyperparameter.

Reviewed July 12, 2026Stable
Reading level: Curious
Pick your depth ↓

When not to use it

  • (You always have one. The question is what breaks.)*
  • Large batch without scaling the learning rate. It'll train worse and you'll blame the batch size.
  • Large batch without warmup. It diverges. This is the source of most "large batches don't work" reports.
  • Tiny batches with BatchNorm. Statistics from 2 examples are noise. Use GroupNorm or LayerNorm.
  • Past critical batch size. More parallelism stops buying speed. You're paying for precision the problem doesn't need.

Reach for something else instead

  • Gradient accumulation — effective large batch on small memory. Slower, and it decouples you from hardware.
  • Gradient checkpointing — trade compute for memory, so a bigger batch fits.
  • LayerNorm/GroupNorm — if small batches are forced on you, remove the BatchNorm dependency.

Further reading

  • Goyal et al. (2017), Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — the linear scaling rule, and why warmup is mandatory with it.
  • Keskar et al. (2016), On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — the influential sharp-minima argument, worth reading alongside its critics.
  • McCandlish et al. (2018), An Empirical Model of Large-Batch Training — critical batch size; the ceiling on useful parallelism.

Primary sources, listed so you can check the claims on this page rather than take them on trust.

Where people go wrong

  • Changing batch size without changing the learning rate. They're coupled roughly linearly.
  • Skipping warmup at large batch. It diverges early and it looks like the batch size is at fault.
  • Repeating "large batches generalise worse" as settled. The sharpness argument has real critics and later work found the penalty largely disappears with retuning.
  • Treating it as a quality knob. It's a hardware consequence you compensate for.

At a glance

FieldDeep Learning
What decides itusually your GPU memory
The couplingdouble batch, double learning rate
Needed at large batchwarmup, or divergence
Contestedthe sharp-minima generalisation story
Hard ceilingcritical batch size
DifficultyBeginner
Flashcards for this concept · study, save or share them →
Question
—
Answer
—
1 / 4

Often compared with

Small vs. large batch — noisy gradients with free regularisation, versus precise gradients that need warmup and a scaled learning rate. Mostly your memory decides, and you compensate.

Where this sits

A destination. 10 concepts lead here, and nothing in the corpus depends on it.

6Levelsteps in
10Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstGradient DescentHyperparameter
LEARN FIRST Gradient Descent Hyperparameter Batch Size
Batch Size sits after Gradient Descent and Hyperparameter, and nothing further depends on it.

Computed from the prerequisite graph, not assigned. How this works

Related concepts

Deep Learning

Learning Rate

How big a step to take when the model updates — the single most important number in training, and the one most people leave at the default.

Deep Learning

Gradient Descent

Walking downhill on the error surface, one small step at a time — how a model's weights actually get updated.

Deep Learning

Batch Normalization

Renormalising activations at every layer — one of deep learning's most important techniques, and its original explanation turned out to be wrong.

Deep Learning

Optimizer

The algorithm that decides how to apply the gradient — where Adam is the default, AdamW is what you should actually use, and SGD still wins sometimes.

Tools & Ecosystem

GPU

The chip that made deep learning possible — thousands of small cores doing the same maths at once, which is exactly what neural networks need.

Deep Learning

Transformer

The neural-network architecture behind almost every modern AI model — built around attention, which lets it weigh every word against every other, all at once.