Why neural networks generalize when theory says they can't

Take a standard image network. Replace every label in the training set with a random one, so no relationship exists between any image and its label. Train it.

It reaches zero training error. It memorises fifty thousand arbitrary assignments perfectly, which is not what a model short of capacity does.

Now restore the real labels and train the same network again. It reaches zero training error, and this time it also classifies images it has never seen.

The second result is the one everyone celebrates. The first is the one that broke the theory, because it establishes that capacity is not what stops a network memorising. The network could memorise. On real data it chose not to, and nothing in classical learning theory says why.

The usual telling is that classical theory was wrong. That is imprecise and it hides the interesting part. The bounds were never violated. They became vacuous, which is a different failure and a more instructive one.

What the classical theory actually says

Worth stating carefully, because the caricature version is what makes the modern result seem paradoxical.

The bias-variance decomposition splits expected error into two competing terms. Bias is error from a model too simple to represent the truth. Variance is error from a model so flexible it fits the noise in this particular sample rather than the structure underneath.

Add capacity and bias falls while variance rises. Somewhere between them is a minimum, and the resulting curve is U-shaped: underfit on the left, overfit on the right, a sweet spot in the middle. Every statistics course teaches this and it is correct.

Vapnik and Chervonenkis made it rigorous. VC dimension measures capacity as the largest set of points a model class can label in every possible way. From it comes a bound: with probability at least 1 minus delta, test error is at most training error plus a term that grows with VC dimension and shrinks with sample size.

That bound is a theorem. It has not been refuted. It is the foundation of statistical learning theory and it is still true today.

Why it stopped being useful

Here is the precise failure, and it is not that the theorem is false.

VC dimension grows with parameter count. A network with a hundred million parameters has enormous VC dimension. Put that into the bound with fifty thousand training examples and the confidence term exceeds one.

The bound then reads: test error is at most 100%.

That is a true statement. It is also worthless, and the technical word for it is vacuous. The theory did not make a wrong prediction; it made no prediction at all. Modern networks live in the regime where the classical bound has nothing to say, and every subsequent development is an attempt to find a bound that stays meaningful there.

This distinction matters because it tells you where to look. A refuted theory needs replacing. A vacuous bound needs a better complexity measure, and the search for one is the whole field.

Double descent: the shape nobody predicted

Then the empirical picture turned out to be stranger than either camp expected.

Plot test error against model size and the classical U appears exactly as taught: error falls, reaches a minimum, then rises as the model starts fitting noise.

Keep going. At the point where the model has just enough capacity to fit the training data exactly, the interpolation threshold, test error spikes. This is the worst possible place to be and it is precisely where the classical theory says you have gone too far.

Push past it. Test error falls again, and in the heavily overparameterised regime it can drop below the classical minimum.

The curve descends twice, and the name stuck. Reported across neural networks, kernel regression, random forests and linear models, so it is a property of overparameterised learning rather than a quirk of deep networks. It also appears along axes other than model size: train for longer and the same shape emerges over epochs.

The practical consequence inverts a standard instinct. A model performing badly may be too small rather than too large, and the fix is more capacity rather than less, which is the opposite of what the U-curve advises.

Benign overfitting: fitting the noise without being harmed by it

Double descent describes what happens. Benign overfitting is the attempt to explain it.

The classical worry about interpolation is specific: a model that fits every training point exactly has fit the noise as well as the signal, and noise does not generalise. That reasoning is sound and it assumes something that turns out to be avoidable.

It assumes fitting the noise and fitting the signal use the same capacity. In high dimensions with enough parameters, they need not. The model can absorb noise into directions that contribute almost nothing to its predictions on new data, while the signal is carried in directions that do. The interpolation is real, the noise is fitted, and the damage is confined.

One formulation calls this a spiky-smooth decomposition: a smooth component that captures the trend, plus narrow spikes that reach individual noisy training points without disturbing the function anywhere else. Interpolation becomes cheap if the spikes are high-frequency and small in the norm that governs generalisation.

The condition for this is not exotic. Given sufficient overparameterisation, almost any reasonable covariance structure permits the minimum-norm interpolating solution to generalise, which is why enormous finite networks work while classical non-parametric theory, reasoning about infinite-dimensional limits, predicted they would fail.

The part that is actually doing the work

If capacity does not determine generalisation, something else does, and the leading answer is uncomfortable for anyone who wanted a clean theory.

The optimiser is part of the model. Among the vast set of parameter settings that fit the training data perfectly, gradient descent does not pick one at random. It converges toward solutions with small norm, and small-norm solutions generalise better. Nobody wrote that preference down; it falls out of the dynamics.

This is implicit regularisation, and it means the thing that controls generalisation is not the architecture but how the architecture was searched. Two networks with identical parameter counts, one trained with stochastic gradient descent and one fitted by a non-gradient method, generalise differently.

Two related observations point the same way. Norm-based bounds replace parameter count with the size of the weight matrices, which stays finite as the network grows and can produce non-vacuous results. And the curvature of the loss surface flattens in the overparameterised regime, so the solutions found are broader minima, and flatness has long been associated with generalisation.

Together these amount to a shift in where complexity is measured. Not in the model class, but in the particular function the training process actually selected.

The three claims, separated

Most confusion here comes from three distinct statements being run together. They have different evidence and different status, and separating them makes the argument tractable.

Claim one: large models can memorise arbitrary data. Established, uncontroversial, demonstrated directly by the random-label experiment. Nobody disputes it.

Claim two: large models nevertheless generalise on real data. Also established, by every working system in production. Also not disputed.

Claim three: we know why. Not established. This is where the disagreement lives, and treating it as settled in either direction is the error.

The reason the first two coexist without contradiction is that they are statements about different things. Claim one is about capacity, what the model class could in principle represent. Claim two is about selection, which function the training process actually produced. Classical theory reasoned entirely about capacity, because for the model classes it was built for the two were tightly coupled: a model that could represent something complicated would tend to.

Overparameterisation decouples them. The class can represent anything, so capacity stops carrying information, and everything now depends on selection. That is the whole shift in one sentence, and it is why the answer keeps turning out to involve the optimiser rather than the architecture.

It also explains why the field's remaining question is so hard. Characterising what a model class can represent is a static problem with mature mathematics behind it. Characterising which function a stochastic, non-convex, high-dimensional optimisation process will land on is a dynamical problem, and the tools for that are much younger.

Grokking, which nobody has fully explained

One phenomenon deserves separate mention because it resists every account above.

Train a network on a small algorithmic task. It reaches perfect training accuracy quickly and test accuracy stays at chance. Keep training long after the training loss has flattened. Then, sometimes after orders of magnitude more steps, test accuracy jumps to near-perfect, suddenly.

The model was memorising, then at some point it found the rule. Nothing in the training loss indicated the transition was coming, and the same phenomenon has been reported on image tasks with convolutional networks and residual networks, so it is not confined to toy problems.

Whatever the eventual theory of generalisation is, it has to explain why a system can sit in a memorising solution indefinitely and then move to a generalising one with no external signal telling it to.

What this means if you are building something

The theory is unsettled and several practical consequences are not.

More capacity is not automatically worse. If your model is underperforming and sits near the interpolation threshold, adding parameters may help rather than hurt. The U-curve intuition actively misleads here.

Zero training error is not evidence of overfitting. In the overparameterised regime, interpolating the training data is normal and compatible with good generalisation. Judge on held-out performance and nothing else.

How you train matters as much as what you train. Since the optimiser selects the solution, changes to the optimiser, learning rate schedule and batch size change generalisation independently of architecture. That is why those choices behave like hyperparameters with outsized effects.

And the ability to memorise is always present. A network that fits real data well could have fit random labels equally well. Nothing structural prevents memorisation, so if your data contains a shortcut, the model may take it, and no capacity argument will stop it.

What is unresolved

There is no accepted theory. Benign overfitting, implicit regularisation, norm-based bounds and flat minima are partial accounts that agree in direction and have not been unified. A 2026 line of work approaches it through broken ergodicity in statistical physics, arguing the dynamics are confined to a subspace determined by initialisation. That it remains open to a new framing indicates how unsettled it is.

Whether double descent has one cause. Recent work on diffusion models found a double descent in distribution space appearing on both training and test metrics simultaneously, which rules out the standard explanation and points to optimisation dynamics instead. The same curve may have different underlying causes in different settings.

Whether any of this transfers to language models. Most of the theory concerns supervised learning on fixed datasets. Whether it describes systems trained once on a corpus approaching the size of the available text, where the notion of a train-test split is itself unclear, is not established.

And why grokking happens at all. The delayed transition from memorisation to generalisation is not predicted by any of the accounts above.

The counter-argument

The classical theory is not obsolete. VC bounds remain correct, remain useful for model classes with bounded capacity, and underpin much of statistics. Presenting deep learning as having overthrown them overstates the case: it found a regime where they say nothing, which is narrower than a refutation.

Double descent may be less universal than the excitement suggests. It requires specific conditions, and some reported instances vanish under different regularisation or data assumptions. Treating it as a law of overparameterised learning is stronger than the evidence supports.

The practical advice is largely unchanged. Hold out data, measure on it, tune against it. Practitioners were doing that before any of this theory existed and would not behave very differently without it. Its value is explanatory rather than operational.

And the theory may be answering a question that is losing relevance. Frontier systems train roughly once, on data whose relationship to any test distribution is unclear. Whether the generalisation framework is the right one for that setting is a fair question, and the honest answer is that nobody knows.

The short version

A network with more parameters than training examples can fit fifty thousand randomly assigned labels perfectly, which establishes that capacity is not what prevents memorisation. On real data the same network interpolates and also generalises, and classical learning theory has no account of why.

The usual claim that classical theory was wrong is imprecise. VC bounds are theorems and remain true. What happened is that VC dimension grows with parameter count, so for a hundred-million-parameter model on fifty thousand examples the bound reads "test error is at most 100%", which is correct and worthless. The bound became vacuous rather than false, and that distinction points at the fix: not a replacement theory, but a better complexity measure.

The empirical picture is stranger than either camp expected. Test error follows the classical U, then spikes at the interpolation threshold where the model just barely fits the data, then descends a second time and can fall below the classical minimum. This double descent appears in neural networks, kernel regression, random forests and linear models, and along epochs as well as model size. The practical inversion: a model performing badly may be too small rather than too large.

Benign overfitting explains how. The classical worry assumes fitting noise and fitting signal consume the same capacity, and in high dimensions with enough parameters they need not: noise can be absorbed into directions that barely affect predictions, described as a smooth component carrying the trend plus narrow spikes reaching individual noisy points.

What actually selects the generalising solution appears to be the optimiser rather than the architecture. Among the many parameter settings that fit the data perfectly, gradient descent converges toward small-norm solutions, and small-norm solutions generalise. Nobody specified that preference; it falls out of the dynamics. Complexity, on this reading, is a property of the function the training process selected, not of the model class it was selected from.

Common questions

Why do overparameterized neural networks generalize? No accepted theory exists, and the leading account is that the optimiser rather than the architecture selects the solution. Among the many parameter settings that fit the training data perfectly, gradient descent converges toward small-norm solutions, and those generalise better. This is called implicit regularisation, and it is supported by norm-based generalisation bounds that stay finite as networks grow, and by the observation that the loss surface flattens in the overparameterised regime.

What is double descent? The observation that test error, plotted against model capacity, falls and rises as classical theory predicts, then spikes at the interpolation threshold where the model just barely fits the training data, then falls a second time in the heavily overparameterised regime, sometimes below the classical minimum. It has been reported in neural networks, kernel regression, random forests and linear models, and also appears over training epochs rather than model size.

What is benign overfitting? Fitting the training data exactly, including its noise, while still generalising well. It is possible because fitting noise and fitting signal need not consume the same capacity: in high dimensions with sufficient parameters, noise can be absorbed into directions that contribute almost nothing to predictions on new data. One formulation describes a smooth component capturing the trend plus narrow spikes reaching individual noisy points without disturbing the function elsewhere.

Is the bias-variance tradeoff wrong? No, and the imprecision matters. The decomposition is a correct mathematical identity and the U-shaped curve describes the underparameterised regime accurately. What fails is the assumption that the curve continues rising indefinitely. Past the interpolation threshold it descends again, so the tradeoff describes part of the picture rather than all of it.

What is VC dimension and why does it not explain deep learning? VC dimension measures a model class's capacity as the largest set of points it can label in every possible way, and yields a bound on test error that grows with capacity and shrinks with sample size. It fails for deep networks because it grows with parameter count, so for a large model on a modest dataset the bound exceeds one and reads "test error is at most 100%". That is true and predicts nothing. The bound is vacuous rather than violated.

Can neural networks memorize random labels? Yes, completely. A standard image network trained on a dataset where every label has been randomised reaches zero training error, memorising tens of thousands of arbitrary assignments. This is the result that broke the classical account, because it establishes that capacity is not what prevents memorisation. The same network on real data also reaches zero training error and generalises, so something other than capacity is doing the work.

What is the interpolation threshold? The point where a model has just enough capacity to fit the training data exactly, driving training error to zero. It is where test error peaks in the double descent curve, making it the worst place to sit, and it separates the classical underparameterised regime from the modern overparameterised one. Being slightly past it is worse than being well past it.

What is grokking? A phenomenon where a network reaches perfect training accuracy with test accuracy at chance, then after far more training, sometimes orders of magnitude more steps, test accuracy jumps suddenly to near-perfect. The model transitions from memorising to having found the rule, with nothing in the training loss signalling the change. Reported on algorithmic tasks and also on image classification, and not predicted by any current account of generalisation.