Artifipedia
GlossaryMapBlog
Home/Deep Learning/Optimizer
Deep Learning

Optimizer

The algorithm that decides how to apply the gradient — where Adam is the default, AdamW is what you should actually use, and SGD still wins sometimes.

Reading level: Curious
Pick your depth ↓

When not to use it

  • (You need one. The question is which.)*
  • Plain Adam, when AdamW exists. The weight decay is being distorted and there's no reason to accept it.
  • Adam when memory is the constraint. Two extra numbers per parameter is a real cost at scale.
  • Adam reflexively on CNNs. SGD with momentum sometimes generalises better there, and uses a third of the memory.
  • A new optimiser from a recent paper. The replication record is poor and the baselines are often under-tuned.

Reach for something else instead

  • AdamW — the answer for nearly everything.
  • SGD + momentum — memory-light, sometimes better generalisation, more tuning.
  • 8-bit Adam / Adafactor — when optimiser state doesn't fit.
  • Lion — less state, competitive, newer and less proven.

Sources & further reading

  • Kingma & Ba (2014), Adam: A Method for Stochastic Optimization — the paper; momentum plus per-parameter scaling.
  • Loshchilov & Hutter (2017), Decoupled Weight Decay Regularization — AdamW; the correction you should be using.
  • Wilson et al. (2017), The Marginal Value of Adaptive Gradient Methods in Machine Learning — adaptive methods can generalise worse than SGD despite lower training loss.

Primary sources, listed so you can check the claims on this page rather than take them on trust.

Where people go wrong

  • Using Adam instead of AdamW out of habit. The weight decay is wrong and the fix is free.
  • Tuning betas. 0.9/0.999 is near-universal; 0.95 for β₂ on LLMs is the one real exception.
  • Forgetting optimiser state in your memory budget. It's roughly 2× the model.
  • Believing "beats Adam" claims without checking whether their Adam was tuned.
  • Assuming adaptive means better. Wilson et al. found it can converge faster to a worse solution.

At a glance

FieldDeep Learning
UseAdamW
Why not Adamits weight decay is distorted by the adaptive scaling
Memory cost~2× model size in optimiser state
Still viableSGD + momentum, especially on CNNs
Research-to-adoption ratiovery poor
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4

Often compared with

AdamW vs. SGD+momentum — one works out of the box on anything and costs 2× memory; the other needs tuning, uses less, and sometimes generalises better.

Related concepts

Deep Learning

Gradient Descent

Walking downhill on the error surface, one small step at a time — how a model's weights actually get updated.

Deep Learning

Learning Rate

How big a step to take when the model updates — the single most important number in training, and the one most people leave at the default.

Deep Learning

Backpropagation

The algorithm that works out which weights caused a mistake and by how much — the reason neural networks can learn at all.

Deep Learning

Loss Function

The number that says how wrong the model is — and therefore the definition of what it's trying to become.

Machine Learning

Regularization

Anything that stops a model fitting the training data too well — a collection of tricks, held together by a story that modern deep learning broke.

Deep Learning

Transformer

The neural-network architecture behind almost every modern AI model — built around attention, which lets it weigh every word against every other, all at once.