Yoshua Bengio

b. 1964 · Quiet progress · 7 works · cited on 6 entries

Diagnosed why deep networks failed to train in 1994, then spent two decades publishing the fixes — including the one that became attention.

Contribution

Bengio's arc is unusually legible: he named the problem before he solved it. Learning Long-Term Dependencies with Gradient Descent is Difficult (1994) explained why recurrent networks forget — gradients vanish through time. Xavier initialisation (2010) and sparse rectifiers (2011) addressed why deep networks failed to train at all. Then Bahdanau, Cho & Bengio (2014) added an alignment mechanism to translation so the decoder could look back at any input position. That mechanism is attention, and the transformer is what happened when someone removed everything around it.

Common misreading

Attention is usually dated to 2017. The mechanism is from 2014, in a machine-translation paper — Attention Is All You Need is a claim about what you can delete, not an introduction of the idea.

Influence

Entries shaped by this work

Deep LearningAttention2014 — the mechanism, three years early Deep LearningVanishing Gradient1994 — named the problem Deep LearningActivation Function2011 — sparse rectifiers Machine LearningHyperparameter2012 — random beats grid Generative AILatent Space2013 — representation learning Applied AIMachine Translation2014 — where attention was born

Selected works

Every reference below links to a search, not a stored URL — so it cannot rot or point at the wrong paper.

Bengio, Y., Simard, P. & Frasconi, P. (1994). Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. cited in the encyclopedia
Glorot, X. & Bengio, Y. (2010). Understanding the Difficulty of Training Deep Feedforward Neural Networks. AISTATS. cited in the encyclopedia
Glorot, X., Bordes, A. & Bengio, Y. (2011). Deep Sparse Rectifier Neural Networks. AISTATS. cited in the encyclopedia
Bergstra, J. & Bengio, Y. (2012). Random Search for Hyper-Parameter Optimization. JMLR, 13, 281–305. cited in the encyclopedia
Bengio, Y., Courville, A. & Vincent, P. (2013). Representation Learning: A Review and New Perspectives. IEEE TPAMI, 35(8), 1798–1828. cited in the encyclopedia
Bahdanau, D., Cho, K. & Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. cited in the encyclopedia
LeCun, Y., Bengio, Y. & Hinton, G. E. (2015). Deep Learning. Nature, 521, 436–444. cited in the encyclopedia