mlsanityData to 14 Oct 2026

Are Transformers universal approximators of sequence-to-sequence functions?

arXiv 1912.10077 ↗First posted 20 Dec 2019cs.LG stat.MLcited by 406

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite the widespread adoption of Transformer models for NLP tasks, the expressive power of these models is not well-understood. In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, which is quite surprising given the amount of shared parameters in these models. Furthermore, using positional encodings, we circumvent the restriction of permutation equivariance, and show that Transformer models can universally approximate arbitrary continuous sequence-to-sequence functions on a compact domain. Interestingly, our proof techniques clearly highlight the different roles of the self-attention and the feed-forward layers in Transformers. In particular, we prove that fixed width self-attention layers can compute contextual mappings of the input sequences, playing a key role in the universal approximation property of Transformers. Based on this insight from our analysis, we consider other simpler alternatives to self-attention layers and empirically evaluate them.

Key facts. Cited by 406 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 37; 2022: 25; 2023: 64; 2024: 97; 2025: 94; 2026: 89.

406 papers.

Sort:

By area:

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  3. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  4. 2022Transformers in Time Series: A Surveycited by 497
  5. 2021Rethinking Graph Transformers with Spectral Attentioncited by 473
  6. 2021Choose a Transformer: Fourier or Galerkincited by 343
  7. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  8. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  9. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
  10. 2023Looped Transformers as Programmable Computerscited by 237
  11. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  12. 2021Thinking Like Transformerscited by 184
38 more
  1. 2023Exphormer: Sparse Transformers for Graphscited by 170
  2. 2023A mathematical perspective on Transformerscited by 164
  3. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  4. 2023The emergence of clusters in self-attention dynamicscited by 140
  5. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 139
  6. 2022On The Computational Complexity of Self-Attentioncited by 139
  7. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
  8. 2023Representational Strengths and Limitations of Transformerscited by 133
  9. 2023What Formal Languages Can Transformers Express? A Surveycited by 124
  10. 2023Teaching Arithmetic to Small Transformerscited by 123
  11. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  12. 2024Large Language Models for Time Series: A Surveycited by 114
  13. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  14. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  15. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  16. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 102
  17. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  18. 2023REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformerscited by 94
  19. 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 93
  20. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  21. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 86
  22. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  23. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  24. 2024Transformers, parallel computation, and logarithmic depthcited by 79
  25. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  26. 2025Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thoughtcited by 77
  27. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  28. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 67
  29. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 67
  30. 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 66
  31. 2024Polynormer: Polynomial-Expressive Graph Transformer in Linear Timecited by 65
  32. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  33. 2025On the Emergence of Position Bias in Transformerscited by 59
  34. 2023Max-Margin Token Selection in Attention Mechanismcited by 58
  35. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  36. 2021MATE: Multi-view Attention for Table Transformer Efficiencycited by 54
  37. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  38. 2023On the Role of Attention in Prompt-tuningcited by 50

50 most cited of 406 All 406 (JSON)