Transformers are Universal Predictors
Not on the map. Why?
Abstract (arXiv)
We find limits to the Transformer architecture for language modeling and show it has a universal prediction property in an information-theoretic sense. We further analyze performance in non-asymptotic data regimes to understand the role of various components of the Transformer architecture, especially in the context of data-efficient training. We validate our theoretical analysis with experiments on both synthetic and real datasets.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 13 of its 26 references that are in the map. First posted on arXiv 15 Jul 2023.
Builds on
13 of 26 references.
References · 13
- 2017Attention Is All You Needcited by 60,361
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2022Emergent Abilities of Large Language Modelscited by 2,512
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
- 2023PaLM 2 Technical Reportcited by 1,162
- 2018Self-Attention with Relative Position Representationscited by 1,120
- 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
- 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
- 2021Pretrained Transformers as Universal Computation Enginescited by 162
- 2020Group Equivariant Stand-Alone Self-Attention For Visioncited by 63
- 2022Constrained Predictive Coding as a Biologically Plausible Model of the Cortical Hierarchycited by 18
1 more
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.