Optimizer

The full account

The proof was wrong

Adam arrived in 2015 and took over almost immediately. The reasons were good: it worked well out of the box, the default hyperparameters were sensible, it needed little tuning, and it came with a convergence proof. That last part mattered — it meant the method wasn't merely convenient but justified.

In 2018, Reddi, Kale and Kumar read the proof carefully. The argument turns on a quantity the paper calls Γt, and the original analysis assumes it is positive semi-definite. For Adam and RMSProp, Γt can be negative. The assumption doesn't hold, so the proof does not establish what it claims. Their paper states it flatly: the proof in the original Adam paper erroneously assumes Γt is positive semi-definite and is hence incorrect.

They didn't stop at finding the hole. They constructed an explicit one-dimensional convex optimisation problem — the easiest possible setting, where every method should work — on which Adam converges to the worst point in the feasible set. Not a slow convergence. The wrong answer, in one dimension, on a convex problem.

The paper won ICLR 2018's best paper award. Three years after the field standardised on Adam, someone checked.

What actually goes wrong

The mechanism is worth understanding because it explains a failure people meet without recognising.

Adam scales each parameter's update by a running estimate of the recent gradient magnitude, held as an exponential moving average of squared past gradients. This is the source of its convenience: parameters with consistently large gradients get smaller steps, and everything self-tunes.

It is also the bug. An EMA forgets. Suppose one rare mini-batch carries a large, highly informative gradient — the sort you get from a rare class, or a rare token in a large output space. Adam takes one step on it, and then the exponential decay washes that gradient out of the average within a few iterations. The information arrives once and is discounted away. Adam gets stuck in its own weighted history, and the rare-but-important signal never accumulates enough influence to move the parameter where it needs to go.

Reddi's fix, AMSGrad, gives the algorithm long-term memory: instead of the EMA, use the maximum of the past squared gradients, so a large gradient's effect on the step size never decays away. It provably fixes the convergence issue, and it solves the synthetic counterexample.

And then nobody used it

Here is the part that makes this page worth writing.

AMSGrad doesn't help in practice. Reddi's own experiments show modest gains, and independent implementations found no meaningful difference on real problems — same validation accuracy on CIFAR-10, same behaviour on logistic regression and feedforward nets. The theoretical defect is real, the counterexample is real, the fix is correct, and on the problems people actually train, it changes nothing you'd notice.

There is also a fair objection to the counterexample itself: contrived one-dimensional problems where an optimiser fails can be constructed for SGD too. A pathological case is a proof that the guarantee is false, not evidence that the failure occurs in your run.

So the field found a genuine hole in the justification for its most-used algorithm, awarded its highest honour to the paper that found it, and then kept using the unjustified algorithm — because it works, and the justified alternative doesn't work better. Eight years later Adam and AdamW are still the default in essentially every deep learning framework.

The improvement people did adopt

Interestingly, the fix that was taken up came from somewhere else entirely and was not about convergence at all.

Loshchilov and Hutter noticed that L2 regularisation and weight decay, which are equivalent under plain SGD, are not equivalent under adaptive methods. Adam's per-parameter scaling divides the L2 gradient term by the same running average as everything else, so the amount of decay a weight receives depends on its gradient history — which is not what anyone intends by weight decay. AdamW decouples them: apply the decay directly to the weights, outside the adaptive scaling.

That change is small, has no bearing on the convergence question, and improved generalisation enough that AdamW replaced Adam as the default almost everywhere. The field ignored the paper that proved its optimiser unjustified and adopted the one that fixed a bookkeeping error in the weight decay.

The other uncomfortable result

Wilson and colleagues made a separate argument in 2017 that has aged well and is still ignored: adaptive gradient methods generalise worse than plain SGD with momentum. They converge faster on the training loss and arrive at solutions that test worse — which is why SGD with momentum, despite requiring real tuning effort, remains the choice for many vision benchmarks and why the biggest language models are trained with careful schedules rather than trusting Adam's adaptivity.

Put together, the honest picture of the field's default optimiser: its convergence proof is wrong, its convergence fix is unused, it generalises worse than the simpler method it replaced, and it is the right choice anyway — because it trains large models reliably with minimal tuning, and that is worth more in practice than any of the above.

What to do

Use AdamW. It is the default for a reason, and the decoupled decay is a genuine improvement.

Don't use AMSGrad because you read about the proof. The proof is broken and the fix does not help you.

Consider SGD with momentum if you have the tuning budget and generalisation is the binding constraint — Wilson's result has held up.

And take the general lesson, which is the same one on the batch normalisation page: a method working is not evidence for the story told about why it works. Adam is a good optimiser and always was. The proof attached to it was wrong for three years, and the algorithm never noticed.