Why self-attention is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries
Not on the map. Why?
Abstract (arXiv)
In this paper, we show that structures similar to self-attention are natural to learn many sequence-to-sequence problems from the perspective of symmetry. Inspired by language processing applications, we study the orthogonal equivariance of seq2seq functions with knowledge, which are functions taking two inputs -- an input sequence and a ``knowledge'' -- and outputting another sequence. The knowledge consists of a set of vectors in the same embedding space as the input sequence, containing the information of the language used to process the input sequence. We show that orthogonal equivariance in the embedding space is natural for seq2seq functions with knowledge, and under such equivariance the function must take the form close to the self-attention. This shows that network structures similar to self-attention are the right structures to represent the target function of many seq2seq problems. The representation can be further refined if a ``finite information principle'' is considered, or a permutation equivariance holds for the elements of the input sequence.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 44 of its 60 references that are in the map. First posted on arXiv 13 Oct 2022.
Builds on
44 of 60 references.
References · 44
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2017Deep Setscited by 1,421
- 2018Self-Attention with Relative Position Representationscited by 1,120
- 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
- 2021E(n) Equivariant Graph Neural Networkscited by 965
32 more
- 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
- 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 746
- 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 607
- 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 441
- 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
- 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
- 2017A Structured Self-attentive Sentence Embeddingcited by 318
- 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
- 2019Cormorant: Covariant Molecular Neural Networkscited by 263
- 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
- 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
- 2017Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanicscited by 214
- 2019Towards Physics-informed Deep Learning for Turbulent Flow Predictioncited by 204
- 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
- 2017Equivariance Through Parameter-Sharingcited by 141
- 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
- 2021Scalars are universal: Equivariant machine learning, structured like classical physicscited by 127
- 2020Incorporating Symmetry into Deep Dynamics Models for Improved Generalizationcited by 126
- 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
- 2021Provably Strict Generalisation Benefit for Equivariant Modelscited by 91
- 2021Learning with invariances in random features and kernel modelscited by 89
- 2018The Description Length of Deep Learning Modelscited by 72
- 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 63
- 2022Your Transformer May Not be as Powerful as You Expectcited by 55
- 2021Rotation Invariant Graph Neural Networks using Spin Convolutionscited by 52
- 2019Physical Symmetries Embedded in Neural Networkscited by 40
- 2020Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?cited by 34
- 2020A Permutation-Equivariant Neural Network Architecture For Auction Designcited by 32
- 2021Universal Approximation Under Constraints is Possible with Transformerscited by 32
- 2020Gauge equivariant neural networks for quantum lattice gauge theoriescited by 28
- 2020CyCNN: A Rotation Invariant CNN using Polar Mapping and Cylindrical Convolution Layerscited by 15
- 2016Permutation-equivariant neural networks applied to dynamics predictioncited by 14
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.