Artifipedia
GlossaryMapBlog
Home/Deep Learning/Learning Rate
Deep Learning

Learning Rate

How big a step to take when the model updates — the single most important number in training, and the one most people leave at the default.

Reading level: Curious
Pick your depth ↓

When not to use it

  • (You always have one. The question is when the default betrays you.)*
  • A constant learning rate on a transformer. No warmup means divergence, often in the first hundred steps.
  • A pretraining learning rate for fine-tuning. 1e-3 on a pretrained model destroys what it knew.
  • The same LR after changing batch size. They're coupled — double the batch, double the rate.
  • Someone else's config, unexamined. It was tuned for their model, their data, their batch size.

Reach for something else instead

  • LR range test — ten minutes, and it just tells you.
  • Learning-rate-free optimisers — adapt the step automatically; genuinely promising.
  • μP — tune on a small model, transfer to the large one. What frontier labs do.
  • Cosine with warmup — the default that works when you don't want to think.

Sources & further reading

  • Smith (2017), Cyclical Learning Rates for Training Neural Networks — the LR range test; ten minutes that beats guessing.
  • Loshchilov & Hutter (2016), SGDR: Stochastic Gradient Descent with Warm Restarts — cosine schedules, now the default.
  • Yang et al. (2022), Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer — μP; tune small, transfer to large.

Primary sources, listed so you can check the claims on this page rather than take them on trust.

Where people go wrong

  • Leaving it at the default and tuning everything else. It's the one that matters most.
  • No warmup on a transformer. It will diverge and you'll blame the architecture.
  • Fine-tuning at pretraining rates. The model gets worse at everything, confusingly.
  • Changing batch size without changing the learning rate, then concluding large batches don't work.
  • Assuming theory can tell you the right value. Seventy years of optimisation theory, and it's still a plot.

At a glance

FieldDeep Learning
Importancethe single most important hyperparameter
How to find itLR range test, ten minutes
Typical1e-3 fresh, 1e-5 fine-tuning an LLM
Coupled withbatch size, roughly linearly
Warmupmandatory for transformers, unexplained
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4

Often compared with

Learning rate vs. every other hyperparameter — this one decides whether training works at all. The rest decide how well.

Related concepts

Deep Learning

Gradient Descent

Walking downhill on the error surface, one small step at a time — how a model's weights actually get updated.

Deep Learning

Optimizer

The algorithm that decides how to apply the gradient — where Adam is the default, AdamW is what you should actually use, and SGD still wins sometimes.

Deep Learning

Batch Size

How many examples the model sees before each update — a systems constraint that everyone treats as a hyperparameter.

Machine Learning

Hyperparameter

A setting you choose rather than learn — and most of the effort spent tuning them goes into the ones that don't matter.

Deep Learning

Loss Function

The number that says how wrong the model is — and therefore the definition of what it's trying to become.

Deep Learning

Transformer

The neural-network architecture behind almost every modern AI model — built around attention, which lets it weigh every word against every other, all at once.