Who invented deep learning, and why it took so long

In 1970 a Finnish student submitted a master's thesis describing an efficient method for computing derivatives through arbitrary networks of operations, with working code. It was written in Finnish. It did not mention neural networks.

Every modern deep learning framework computes gradients using that method.

Seppo Linnainmaa's reverse-mode automatic differentiation is backpropagation, arrived at from numerical analysis rather than from learning. He was first, and almost nobody in machine learning read it, and the field went on to discover the same algorithm repeatedly over the following fifteen years.

Backpropagation was invented at least four times. Convolutional architecture arrived in 1979. By 1990 nearly every component of modern deep learning existed and had been published. The twenty-year delay was not caused by anyone failing to have the idea, which makes the usual story about a breakthrough moment substantially wrong.

Priority and paternity are different things

The cleanest framing for this comes from a distinction that resolves most of the credit argument.

Priority is being first. Paternity is being the version everyone descends from. They frequently attach to different people, and the backpropagation story separates them unusually clearly.

Linnainmaa, 1970: priority without paternity. A master's thesis in Finnish with FORTRAN code, framed as numerical analysis, never connected to networks. Correct, first, and outside the conversation.

Werbos, 1974: priority and paternity. A Harvard dissertation proposing that reverse-mode differentiation could train neural networks, which is the connection Linnainmaa did not make. He then spent years unable to publish it. The first winter was under way, neural network research was out of favour, and the reception to a paper about training neural networks in the mid-1970s was predictable. It reached print around 1981.

Rumelhart, Hinton and Williams, 1986: paternity without priority. They reinvented it, and Rumelhart later said plainly he had not cited the earlier work because it was obscure enough that he did not know of it. Hinton has confirmed the same, without much concern about the dispute. Their paper is why nobody needed to invent it again.

The lesson usually drawn is about credit and the useful one is about mechanism. It is generally not the first inventor who is remembered, but the last reinventor, because what makes an idea stick is arriving when the field is ready to use it.

Four inventions, four receptions

The dates alone understate it. What separates the versions is not the mathematics, which is close to identical, but the circumstances each arrived into.

1970, Linnainmaa. Correct, first, complete with code. Written in Finnish, framed as numerical analysis, published where numerical analysts read. The machine learning community had no reason to encounter it and no vocabulary that would have made it recognisable if they had. Wrong field, wrong language, right answer.

1971 to 1974, Werbos. The right connection, made to the right audience, at the worst possible moment. Neural networks were the losing side of a paradigm dispute and about to lose their funding entirely. A dissertation arguing that multilayer networks could be trained was, in 1974, an argument that the field had just collectively decided against. Right answer, right field, wrong decade.

Early 1980s, various. Several independent rediscoveries in adjacent areas including control theory. Each solved its own problem and did not generalise outward, because nobody was looking across the boundary. Right answer, too narrow a frame.

1986, Rumelhart, Hinton and Williams. Arriving after expert systems had revived interest in AI generally, into a community with computers capable of running the demonstrations, alongside a result people found persuasive for reasons beyond the algorithm: hidden layers learning interpretable internal representations. Right answer, and the first time anything was in place to receive it.

The through-line is that three of the four failures were failures of context rather than content. An idea needs a field ready to use it, a vocabulary that makes it legible, and machinery that lets someone demonstrate it. Backpropagation had none of those in 1970 and all three in 1986, and the mathematics did not change.

What else was already there

Backpropagation is the famous case. The pattern is broader, and the dates are worth seeing together.

1949, Hebbian learning. Hebb proposed that connections strengthen when units activate together, giving the first mechanism for learning in a network of simple elements.

1958, the perceptron. Rosenblatt built a learning machine that adjusted its own weights, with a convergence proof for the linearly separable case.

1979 to 1980, the neocognitron. Fukushima built a hierarchical multilayer network with local receptive fields and alternating layers of simple and complex cells, inspired by visual cortex. This is a convolutional neural network, a decade before the term. It lacked an efficient training method, which is precisely the gap backprop filled.

1989, LeNet. LeCun paired convolutional architecture with backpropagation training and applied it to handwritten digits, ultimately reading bank cheques in production. Convolution, feature hierarchy and weight sharing, all working, all deployed.

By the end of the 1980s the field had the architecture, the training algorithm and a working commercial application. It then waited more than twenty years for widespread acceptance.

What was actually missing

Not the ideas. The argument that has held up best identifies the constraint as hardware, and the framing is worth taking seriously because it generalises.

Research directions succeed partly by fitting the hardware that happens to exist. During the era when general-purpose CPUs were what everyone had, approaches that suited them prospered and approaches that needed dense parallel arithmetic did not. Neural networks needed the second kind of machine, and the machine arrived from an unrelated industry.

Graphics processors were built to shade pixels. They turned out to be dense matrix multipliers, which is what training a network mostly consists of. Nobody designed them for this, and the field that had been waiting twenty years for suitable hardware got it as a side effect of video games.

The uncomfortable implication: an idea's success depends partly on a hardware accident, and there is no reason to think the current accident favours the best available idea. Approaches that would work well on hardware nobody builds are invisible, and they will stay invisible.

Three other constraints mattered alongside it. Labelled data at scale did not exist until the web produced it. Practical techniques for training deep stacks, including better initialisation, normalisation and activation functions, took years to accumulate. And a decade of institutional disfavour meant fewer people working on it, which slows everything.

Why this history is not just history

Three things follow that apply directly to reading current work.

A published idea is not a used idea. Werbos had the right answer in 1974 and could not get it into print, and Linnainmaa's method sat in a Finnish thesis while the field rediscovered it twice. If the constraint on progress is frequently attention rather than insight, then a great deal of currently published work is in the same position, and nobody knows which parts.

Credit tracks adoption rather than invention, which distorts the record in a predictable direction. The 1986 paper is cited orders of magnitude more than the 1974 dissertation. That is not dishonesty; it is what citation measures. But treating citation counts as a measure of originality reads the wrong quantity.

And the hardware question is live. Current architectures are shaped by what runs well on GPUs, which is dense parallel arithmetic on tensors. Symbolic reasoning, sparse computation and sequential algorithms all sit outside that, which is one reason they look uncompetitive. Some of that reflects genuine inferiority and some reflects the same accident that suppressed neural networks for twenty years.

What is unresolved

Whether the delay was avoidable. The counterfactual assumes that with better communication and less institutional hostility, the field arrives sooner. It is equally possible the ideas needed the hardware regardless, in which case the winter cost less than it appears.

How much of the modern era is new. Architecturally, transformers are truly different from what existed in 1990. Whether the last fifteen years represent a comparable density of new ideas or mostly the execution of old ones at scale is arguable, and the answer differs by subfield.

What is currently sitting unread. The uncomfortable corollary of the whole story is that if it happened before, it is probably happening now, and by construction nobody can name the examples.

And whether the hardware lottery has a winner worth having. The current one favours dense parallel arithmetic. Whether that is the right substrate for the problems that remain, or simply the substrate that exists, is not something the field can answer from inside it.

The counter-argument

Priority disputes are frequently overstated. Linnainmaa did not connect his method to learning, and Werbos's version was not the practical training procedure that made deep networks work. The 1986 paper contained real contributions beyond the algorithm, including the demonstration that hidden layers learn useful internal representations, which was the finding that persuaded people.

The hardware argument can be too tidy. It is a satisfying single explanation for a messy period that also involved funding, fashion, missing data, unsolved optimisation problems and a shortage of people. Attributing the delay primarily to hardware makes a good story and probably overweights one factor.

And "the ideas were already there" flatters hindsight. They were there among many other ideas that did not work, without any way to tell them apart. Knowing which of the available directions would eventually succeed is the entire difficulty, and it is invisible from the far side.

The short version

A 1970 Finnish master's thesis described reverse-mode automatic differentiation with working code, without mentioning neural networks. Every modern framework computes gradients that way. Werbos connected the method to network training in a 1974 dissertation and could not get it published for years, since neural network research was out of favour during the first winter. Rumelhart, Hinton and Williams reinvented it in 1986 without knowing of the earlier work, and their paper is why nobody needed to invent it again.

Priority and paternity are different. Linnainmaa had priority without paternity, Werbos had both and was ignored, Rumelhart had paternity without priority. It is generally not the first inventor who is remembered but the last reinventor, because what makes an idea stick is arriving when the field can use it.

The pattern is broader than backpropagation. Hebbian learning in 1949, the perceptron in 1958, Fukushima's neocognitron in 1979 with local receptive fields and alternating simple and complex cells, which is a convolutional network a decade before the name, lacking only an efficient training method. LeCun paired convolution with backpropagation in 1989 and put it into production reading bank cheques.

By 1990 the field had the architecture, the training algorithm and a deployed commercial application, and then waited more than twenty years for acceptance. What was missing was not insight. Research directions succeed partly by fitting the hardware that happens to exist, and neural networks needed dense parallel arithmetic that arrived as a side effect of the video game industry.

The corollary is the part worth carrying: an idea's success depends partly on a hardware accident, and there is no reason to think the current accident favours the best available idea. Approaches that would work well on hardware nobody builds are invisible, and they stay invisible.

Common questions

Who invented backpropagation? Several people, independently. Seppo Linnainmaa described the method in a 1970 master's thesis as reverse-mode automatic differentiation, with FORTRAN code and no mention of neural networks. Paul Werbos connected it to training networks in a 1974 Harvard dissertation and struggled to publish for years. Rumelhart, Hinton and Williams reinvented it and published in 1986, which is the version that stuck. Hinton has confirmed it was independently invented by many people before that paper.

Who invented deep learning? No single person, and the components arrived across four decades. Hebb proposed a learning mechanism in 1949, Rosenblatt built the perceptron in 1958, Fukushima built the first convolutional architecture in 1979, backpropagation was invented repeatedly through the 1970s and popularised in 1986, and LeCun combined convolution with backpropagation in 1989. By 1990 the essential pieces existed and had been published.

What was the neocognitron? A hierarchical multilayer neural network built by Kunihiko Fukushima around 1979, inspired by the visual cortex, using local receptive fields and alternating layers of simple and complex cells. It is a convolutional neural network in all but name, predating the term by roughly a decade. What it lacked was an efficient training method, which is exactly the gap backpropagation filled once the two were combined.

Why did deep learning take so long to work? Not because the ideas were missing. The most durable explanation is hardware: research directions succeed partly by fitting the machines that happen to exist, and during the general-purpose CPU era the approaches suited to CPUs prospered while neural networks, which need dense parallel arithmetic, did not. Graphics processors built for video games turned out to be matrix multipliers. Labelled data at scale, better training techniques and institutional support were also missing.

What is the hardware lottery? The observation that an idea's success depends partly on whether it happens to suit the hardware available at the time, rather than purely on merit. Deep learning is the standard example: the components existed for decades and were widely dismissed until hardware from an unrelated industry made them practical. The uncomfortable implication is that approaches suited to hardware nobody builds remain invisible, and there is no way to identify them from inside the current regime.

Why is Rumelhart credited if he was not first? Because credit in science tracks adoption rather than invention. His 1986 paper was the version the field read, understood and built on, and it contained contributions beyond the algorithm itself, notably the demonstration that hidden layers learn useful internal representations. A useful distinction is between priority, meaning being first, and paternity, meaning being the version everyone descends from. They frequently attach to different people.

When were convolutional neural networks invented? Around 1979, by Kunihiko Fukushima, as the neocognitron. It had the hierarchical structure, local receptive fields and the alternating simple and complex cell layers that define the architecture. It could not be trained efficiently. LeCun's 1989 work paired the same architecture with backpropagation, producing LeNet, which read handwritten digits in production. The core principles of convolution, feature hierarchy and weight sharing date from that period and remain in use.

Were the ideas behind deep learning available before 2012? Almost all of them. Architecture, training algorithm and a deployed commercial application existed by 1990. What changed in the following two decades was the arrival of suitable parallel hardware, labelled data at web scale, an accumulation of practical training techniques including better initialisation and normalisation, and enough institutional confidence to fund the work. The 2012 result is better understood as the moment the constraints lifted than as the moment the ideas appeared.