Artifipedia
GlossaryMapBlog
Home/Deep Learning/Batch Size
Deep Learning

Batch Size

How many examples the model sees before each update — a systems constraint that everyone treats as a hyperparameter.

Reading level: Curious
Pick your depth ↓

When not to use it

  • (You always have one. The question is what breaks.)*
  • Large batch without scaling the learning rate. It'll train worse and you'll blame the batch size.
  • Large batch without warmup. It diverges. This is the source of most "large batches don't work" reports.
  • Tiny batches with BatchNorm. Statistics from 2 examples are noise. Use GroupNorm or LayerNorm.
  • Past critical batch size. More parallelism stops buying speed. You're paying for precision the problem doesn't need.

Reach for something else instead

  • Gradient accumulation — effective large batch on small memory. Slower, and it decouples you from hardware.
  • Gradient checkpointing — trade compute for memory, so a bigger batch fits.
  • LayerNorm/GroupNorm — if small batches are forced on you, remove the BatchNorm dependency.

Sources & further reading

  • Goyal et al. (2017), Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — the linear scaling rule, and why warmup is mandatory with it.
  • Keskar et al. (2016), On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — the influential sharp-minima argument, worth reading alongside its critics.
  • McCandlish et al. (2018), An Empirical Model of Large-Batch Training — critical batch size; the ceiling on useful parallelism.

Primary sources, listed so you can check the claims on this page rather than take them on trust.

Where people go wrong

  • Changing batch size without changing the learning rate. They're coupled roughly linearly.
  • Skipping warmup at large batch. It diverges early and it looks like the batch size is at fault.
  • Repeating "large batches generalise worse" as settled. The sharpness argument has real critics and later work found the penalty largely disappears with retuning.
  • Treating it as a quality knob. It's a hardware consequence you compensate for.

At a glance

FieldDeep Learning
What decides itusually your GPU memory
The couplingdouble batch, double learning rate
Needed at large batchwarmup, or divergence
Contestedthe sharp-minima generalisation story
Hard ceilingcritical batch size
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4

Often compared with

Small vs. large batch — noisy gradients with free regularisation, versus precise gradients that need warmup and a scaled learning rate. Mostly your memory decides, and you compensate.

Related concepts

Deep Learning

Learning Rate

How big a step to take when the model updates — the single most important number in training, and the one most people leave at the default.

Deep Learning

Gradient Descent

Walking downhill on the error surface, one small step at a time — how a model's weights actually get updated.

Deep Learning

Batch Normalization

Renormalising activations at every layer — one of deep learning's most important techniques, and its original explanation turned out to be wrong.

Deep Learning

Optimizer

The algorithm that decides how to apply the gradient — where Adam is the default, AdamW is what you should actually use, and SGD still wins sometimes.

Tools & Ecosystem

GPU

The chip that made deep learning possible — thousands of small cores doing the same maths at once, which is exactly what neural networks need.

Deep Learning

Transformer

The neural-network architecture behind almost every modern AI model — built around attention, which lets it weigh every word against every other, all at once.