Regularization: L1, L2, dropout and what unites them

Add a penalty on the squared size of your weights and you have applied L2 regularization. Assume before seeing any data that weights are drawn from a Gaussian centred at zero, then find the most probable weights given the data, and you have done Bayesian MAP estimation.

These are the same computation. Not similar, not analogous. Minimising the L2-penalised loss is provably identical to maximum a posteriori estimation under a Gaussian prior, and the penalty coefficient is exactly how strongly you hold that prior against the evidence.

L1 has the same correspondence with a Laplace prior. Once you see it, the rest of the list stops looking like a list.

Regularization is not a collection of tricks for preventing overfitting. It is the expression of a belief about which functions are more likely, imposed from outside the training data. Six techniques taught as separate topics are six ways of writing down a prior, and the differences between them are differences of belief rather than of mechanism.

The six, and what each one believes

L2, or weight decay. Penalise the sum of squared weights. The belief: small weights are more likely than large ones. Geometrically the penalty draws concentric circles in weight space and the solution is pulled toward the origin. Weights shrink toward zero and rarely reach it, because the penalty subtracts a fraction of the current weight each step, and a fraction of a small number is smaller still.

L1, or lasso. Penalise the sum of absolute values. The belief: most weights should be exactly zero. The mechanism for that is worth understanding rather than memorising. L1 subtracts a roughly constant amount each step regardless of current size, so a small weight can be driven to exactly zero and stay there. Geometrically the constraint region is a diamond with corners on the axes, and a contour touching a corner means a coordinate is exactly zero. Sparsity is a consequence of the geometry.

Dropout. Randomly zero units during training. The belief: no single unit should be load-bearing. The theoretical reading is stronger than that: with N units, dropout implicitly trains something like 2^N thinned sub-networks that share weights, and using all units at test time approximates averaging their predictions. It is an ensemble you did not pay for.

Early stopping. Halt training before convergence. The belief: the functions reachable in fewer steps are more likely. This is a constraint on the norm of the solution imposed through the optimisation path rather than through the loss, and for simple models the correspondence with L2 is exact.

Data augmentation. Train on transformed copies. The belief: these transformations should not change the label. A rotated cat is a cat. That is a statement about the world, expressed as data rather than as a penalty, and it is the most explicit prior on the list because you have to name the invariance.

Batch normalisation. Normalise activations using batch statistics. It was not designed as a regulariser and acts as one, because statistics computed over a random mini-batch are noisy, and noise during training is itself a constraint on what can be learned.

Six mechanisms, six beliefs. The choice between them is a choice about what you think is true, and nothing about the mathematics tells you which belief is correct.

Why the Bayesian reading is worth having

Not because you should become a Bayesian. Because it makes the hyperparameter interpretable.

The regularization coefficient is usually treated as a number to be tuned by search. Under the prior reading it has a meaning: it is the strength of your belief relative to the evidence. A large coefficient says the prior should dominate unless the data argues loudly. A small one says the data should mostly win.

That reframes the tuning problem. Sweeping the coefficient is asking how much you should trust your assumptions given this dataset, and the answer depends on how much data you have, which is why the best coefficient shrinks as datasets grow. More evidence should overrule more prior. The search is discovering something rather than fitting a knob.

It also explains why the techniques compose the way they do. Practical guidance converges on combining techniques from different families rather than stacking several from one, which under this reading is obvious: two penalties on weight magnitude are two versions of the same belief, while weight decay plus augmentation are separate claims about separate things.

The one thing no regularizer can tell you

Every technique above is a bet, and the bets are not neutral. This is the part that gets left out of the tutorials.

A preference for smooth functions is correct only if the truth is smooth. L2 believes small weights are more likely. That belief is right for most natural signals and wrong for a problem whose true structure involves a few very large effects. Applying it there does not fail loudly; it produces a model that has quietly averaged away the thing that mattered.

A preference for sparsity is correct only if most features are irrelevant. L1 is excellent when 200 undocumented features contain 30 useful ones, and actively harmful when the truth is a broad combination of weak signals, because it will pick a few and zero the rest.

A belief that a transformation preserves the label is correct only when it does. Rotating a cat gives a cat. Rotating a 6 gives a 9, and a digit classifier trained with rotation augmentation has been told something false about its own domain.

There is no universally correct regularizer, and this is not a practical inconvenience but a theorem. Any preference that helps on one class of problems must hurt on another, because the preference is doing work only where it excludes something, and what it excludes is sometimes the answer.

Which makes the choice a domain question rather than a tuning question. The coefficient can be found by search. Whether to believe in smoothness or sparsity or rotational invariance cannot, because the data cannot tell you what to assume before you have assumed something. That is the part that requires knowing what you are modelling, and it is the part most consistently skipped.

What the modern picture broke

The classical account is that regularization prevents overfitting by restricting capacity. That account has a problem, and it is the same result that broke the classical theory of generalization.

Networks with regularization still fit random labels perfectly. Weight decay on, dropout on, and the model memorises fifty thousand arbitrary assignments regardless. So whatever regularization is doing, it is not preventing memorisation, because the capacity to memorise survives it intact.

The modern reading changes the job description rather than discarding the technique. Explicit regularization does not restrict which functions are reachable. It expresses a preference among the many functions that fit the data equally well, in a regime where there are always many.

That puts it alongside the implicit regularization of the optimiser, which does the same job without being asked. Gradient descent converges toward small-norm solutions among all those that interpolate, which is a preference nobody wrote down, and it appears to do more work than the explicit penalty added on top.

The uncomfortable version: the coefficient you tune may be a small correction to a much larger effect you do not control.

The failure mode nobody warns about

Worth stating because it looks like success.

Regularization pushes a model toward smooth, robust solutions. When the training signal is truly inconsistent, that is exactly the wrong thing, and the result is not visible in the usual metrics.

One documented case: a content moderation system with a 15% disagreement rate among its human labels. Aggressive dropout and weight decay produced good validation numbers and a model reporting 92% confidence on controversial posts, while splitting evenly between allow and remove based on trivial phrasing differences. It had learned a smooth interpolation between contradictory labels.

That is worse than overfitting, because overfitting is visible. A model that memorised the contradictions would at least have shown poor validation performance. This one looked like successful generalisation and was confidently arbitrary.

The rule that follows: regularization assumes there is a coherent signal to smooth toward. Before turning it up, check whether the labels agree with each other, because forcing robustness onto noise produces confident inconsistency rather than caution.

What is unresolved

Whether the Bayesian correspondence is meaningful for deep networks. The identity between L2 and a Gaussian prior is exact for the mathematics. Whether it is interpretable here is disputed, since the magnitude of a weight vector feeding a scale-invariant layer does not affect the network's function at all, which makes the prior's variance meaningless in those cases. One reading treats weight decay in deep networks as a form of normalisation rather than a prior.

How much explicit regularization matters at scale. In the overparameterised regime, implicit regularization from the optimiser appears to dominate. Whether explicit penalties are a meaningful contribution or a small correction is setting-dependent and not well characterised.

Whether the taxonomy survives. Recent surveys note that standard treatments predate double descent and grokking, and that those findings reshaped the understanding of generalisation without the practical guidance being rewritten to match. The techniques still work; the explanation for why has changed underneath them.

And whether any of this transfers to models trained once on a corpus. Nearly all of the theory concerns repeated training on a fixed dataset with a held-out set. What weight decay is doing in a single pass over a trillion tokens is a different question and gets asked less than it should.

The counter-argument

The unification can be overstated. Saying everything is a prior is tidy and it does not help you choose. Dropout and weight decay behave very differently in practice, dropout works better on fully connected layers than on convolutional or attention layers, and no amount of Bayesian framing predicts that. The mechanisms differ in ways that matter operationally even if they share an interpretation.

The prior reading is a post-hoc rationalisation for several of them. Batch normalisation was not designed as a regulariser, early stopping was a practical expedient, and dropout was motivated by an analogy to sexual reproduction rather than by ensembling. Reading them all as principled statements of belief tidies a history that was mostly empirical.

And practitioners do not need it. The operational advice, combine techniques from different families, tune on validation data, reduce regularization as data grows, is derivable from experiment without any theory. The unification is intellectually satisfying and changes very little about what anyone does on Monday.

The short version

Minimising an L2-penalised loss is provably identical to maximum a posteriori estimation with a Gaussian prior on the weights, with the penalty coefficient being exactly how strongly the prior is held against the evidence. L1 corresponds to a Laplace prior in the same way. The correspondence is an identity rather than an analogy, and once seen it reorganises the standard list of techniques.

Six methods, six beliefs. L2 believes small weights are more likely. L1 believes most weights should be exactly zero, and produces sparsity because it subtracts a constant amount each step rather than a fraction, so a weight can actually reach zero. Dropout believes no unit should be load-bearing, and implicitly trains something like 2^N weight-sharing sub-networks. Early stopping believes functions reachable in fewer steps are more likely. Data augmentation believes stated transformations do not change the label. Batch normalisation regularises by accident, through noise in batch statistics.

The Bayesian reading makes the hyperparameter interpretable: it is the strength of belief relative to evidence, which is why the best coefficient shrinks as datasets grow, and why combining techniques from different families beats stacking several from one.

The modern picture changed the job rather than the technique. Networks with regularization on still fit random labels perfectly, so regularization is not what prevents memorisation. It expresses a preference among the many functions that fit equally well, alongside the implicit regularization of the optimiser, which appears to do more of the work.

And the failure mode that looks like success: regularization assumes a coherent signal to smooth toward. Applied to truly inconsistent labels it produces confident arbitrariness, as in a moderation system that reported 92% confidence while splitting evenly on trivial phrasing differences. Overfitting is at least visible. This is not.

Common questions

What is regularization in machine learning? Any technique that expresses a preference for some solutions over others, imposed from outside the training data. Classically it was described as restricting model capacity to prevent overfitting. The more accurate modern description is that it selects among the many functions that fit the data equally well, since in overparameterised models there are always many, and capacity restriction is not what is happening.

What is the difference between L1 and L2 regularization? L2 penalises squared weights and shrinks them toward zero without reaching it, because it subtracts a fraction of the current weight each step. L1 penalises absolute values and drives weights to exactly zero, because it subtracts a roughly constant amount regardless of size. Geometrically, L2's constraint region is a circle and L1's is a diamond with corners on the axes, and touching a corner means a coordinate is exactly zero. That is why L1 produces sparsity and does feature selection.

Is weight decay the same as L2 regularization? In classical gradient descent they are equivalent, and weight decay is the name for the update-rule form where each weight is multiplied by slightly less than one every step. With adaptive optimisers the two come apart, because the adaptive scaling interacts with the penalty, which is why decoupled weight decay was introduced as a separate mechanism. In practice they are used interchangeably and the distinction matters when using Adam.

What is dropout and why does it work? Randomly zeroing units during training so no single unit becomes load-bearing. The stronger theoretical reading is that with N units it implicitly trains something like 2^N thinned sub-networks sharing weights, and using all units at test time approximates averaging their predictions, which makes it an ensemble obtained for free. It works better on fully connected layers than on convolutional layers, where spatial correlations are expected, or attention layers, where consistency is wanted.

Is early stopping a form of regularization? Yes. Halting before convergence constrains which solutions are reachable, expressing a belief that functions found in fewer steps are more likely to generalise. For simple models the correspondence with L2 regularization is exact, with the number of steps playing the role of the penalty coefficient. It is regularization imposed through the optimisation path rather than through the loss function.

Does regularization prevent memorization? No, and this is one of the results that reshaped the theory. Networks with weight decay and dropout applied still fit randomly assigned labels perfectly, memorising tens of thousands of arbitrary assignments. Whatever regularization does, the capacity to memorise survives it. The modern reading is that it expresses a preference among functions that all fit the data, rather than removing the ability to fit anything.

How do I choose which regularization to use? Combine techniques from different families rather than stacking several from one, since two penalties on weight magnitude are two versions of the same belief while weight decay plus data augmentation are separate claims. Reduce the coefficient as your dataset grows, because it represents belief strength relative to evidence and more evidence should overrule more prior. And check label consistency before turning it up, since regularization assumes a coherent signal to smooth toward.

When does regularization make things worse? When the training signal is truly inconsistent. Regularization pushes toward smooth robust solutions, and forcing that onto contradictory labels produces a model that confidently interpolates between them. One documented case reported 92% confidence on controversial content while splitting evenly between opposite decisions based on trivial phrasing. This is worse than overfitting because overfitting shows up as poor validation performance, while this looks like successful generalisation.