A Probabilistic Interpretation of Transformers
Not on the map. Why?
Abstract (arXiv)
We propose a probabilistic interpretation of exponential dot product attention of transformers and contrastive learning based off of exponential families. The attention sublayer of transformers is equivalent to a gradient ascent step of the log normalizer, which is the log-sum-exp term in the Hopfield theory of attention. This ascent step induces a parallel expansion of points, which is counterbalanced by a contraction from layer normalization. We also state theoretical limitations of our theory and the Hopfield theory and suggest directions for resolution.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 15 of its 32 references that are in the map. First posted on arXiv 28 Apr 2022.
Builds on
15 of 32 references.
References · 15
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
- 2021Convex optimizationcited by 2,515
- 2014Neural Turing Machinescited by 644
- 2019Deep Equilibrium Modelscited by 615
- 2013Generating Sequences With Recurrent Neural Networkscited by 554
- 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
- 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
3 more
- 2015Neural GPUs Learn Algorithmscited by 121
- 2020Monotone operator equilibrium networkscited by 100
- 2018Can recurrent neural networks warp time?cited by 55
Built on by
Citing papers per year: 2025: 1.
1 paper.