mlsanityData to 14 Oct 2026

Approximation Rate of the Transformer Architecture for Sequence Modeling

arXiv 2305.18475 ↗First posted 29 May 2023cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

The Transformer architecture is widely applied in sequence modeling applications, yet the theoretical understanding of its working principles remains limited. In this work, we investigate the approximation rate for single-layer Transformers with one head. We consider a class of non-linear relationships and identify a novel notion of complexity measures to establish an explicit Jackson-type approximation rate estimate for the Transformer. This rate reveals the structural properties of the Transformer and suggests the types of sequential relationships it is best suited for approximating. In particular, the results on approximation rates enable us to concretely analyze the differences between the Transformer and classical sequence modeling methods, such as recurrent neural networks.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

21 of 37 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  5. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  6. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  7. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  8. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  9. 2023Looped Transformers as Programmable Computerscited by 237
  10. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  11. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 96
  12. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 89
9 more
  1. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  2. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 83
  3. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  4. 2020The Depth-to-Width Interplay in Self-Attentioncited by 42
  5. 2021Universal Approximation Under Constraints is Possible with Transformerscited by 32
  6. 2020On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysiscited by 27
  7. 2021Can Vision Transformers Perform Convolution?cited by 18
  8. 2021On the rate of convergence of a classifier based on a Transformer encodercited by 16
  9. 2021Approximation Theory of Convolutional Architectures for Time Series Modellingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 8; 2026: 12.

28 papers.

Sort:

By area:

  1. 2023A mathematical perspective on Transformerscited by 164
  2. 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 47
  3. 2024On Statistical Rates and Provably Efficient Criteria of Latent Diffusion Transformers (DiTs)cited by 40
  4. 2024On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encodingcited by 36
  5. 2024Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modelingcited by 25
  6. 2024On the Optimal Memorization Capacity of Transformerscited by 10
  7. 2024CARROT: A Learned Cost-Constrained Retrieval Optimization System for RAGcited by 6
  8. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5
  9. 2025Approximation Bounds for Transformer Networks with Application to Regressioncited by 4
  10. 2025The Effect of Attention Head Count on Transformer Approximationcited by 4
  11. 2026Affine-Scaled Attention: Towards Flexible and Stable Transformer Attentioncited by 3
  12. 2025A unified framework for establishing the universal approximation of transformer-type architecturescited by 2
16 more
  1. 2026Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unitycited by 2
  2. 2026Measure-to-measure Regression with Transformerscited by 2
  3. 2024How Out-of-Distribution Detection Learning Theory Enhances Transformer: Learnability and Reliabilitycited by 1
  4. 2025Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functionscited by 1
  5. 2025Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regimecited by 1
  6. 2026LLM-Inspired Pretrain-Then-Finetune for Small-Data, Large-Scale Optimizationcited by 1
  7. 2026On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functionscited by 1
  8. 2026Training-Free Universal Approximation by Prompting Random Transformerscited by 1
  9. 2026Universal Approximation of Measure-to-Measure Operators by Pushforwardscited by 1
  10. 2025Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspectivecited by 0
  11. 2025On the Universality of Transformer Architectures; How Much Attention Is Enough?cited by 0
  12. 2026Position: Agentic AI System Is a Foreseeable Pathway to AGIcited by 0
  13. 2026InfoFlow: A Framework for Multi-Layer Transformer Analysiscited by 0
  14. 2026How Many Different Outputs Can a Transformer Generate?cited by 0
  15. 2026Foundation-Preserving Optimization in Generalized Eigenspacecited by 0
  16. 2026Dualformer: Efficient Feature Extractor for Complex-valued Blind Communication Signal Analysiscited by 0