mlsanityData to 7 Oct 2026

Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input

arXiv 2305.18699 ↗First posted 30 May 2023cs.LG stat.MLcited by 32

Not on the map. Why?

Abstract (arXiv)

Despite the great success of Transformer networks in various applications such as natural language processing and computer vision, their theoretical aspects are not well understood. In this paper, we study the approximation and estimation ability of Transformers as sequence-to-sequence functions with infinite dimensional inputs. Although inputs and outputs are both infinite dimensional, we show that when the target function has anisotropic smoothness, Transformers can avoid the curse of dimensionality due to their feature extraction ability and parameter sharing property. In addition, we show that even if the smoothness changes depending on each input, Transformers can estimate the importance of features for each input and extract important features dynamically. Then, we proved that Transformers achieve similar convergence rate as in the case of the fixed smoothness. Our theoretical results support the practical success of Transformers for high dimensional data.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

22 of 34 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  9. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  10. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
  11. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  12. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
10 more
  1. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  2. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  3. 2022Vision Transformers provably learn spatial structurecited by 92
  4. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
  5. 2018Deep Neural Networks Learn Non-Smooth Functions Effectivelycited by 57
  6. 2019Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov spacecited by 46
  7. 2021On the Expressive Power of Self-Attention Matricescited by 44
  8. 2019On the minimax optimality and superiority of deep neural network learning over sparse parameter spacescited by 24
  9. 2021On the rate of convergence of a classifier based on a Transformer encodercited by 16
  10. 2022An Analysis of Attention via the Lens of Exchangeability and Latent Variable Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 12; 2026: 9.

32 papers.

Sort:

By area:

  1. 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 46
  2. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 36
  3. 2024Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularitycited by 33
  4. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
  5. 2024On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encodingcited by 28
  6. 2024What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Taskscited by 19
  7. 2024How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regressioncited by 13
  8. 2025Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methodscited by 13
  9. 2024On the Optimal Memorization Capacity of Transformerscited by 10
  10. 2024State Space Models are Provably Comparable to Transformers in Dynamic Token Selectioncited by 5
  11. 2024Global Convergence in Training Large-Scale Transformerscited by 5
  12. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5
20 more
  1. 2025Hallucinations are inevitable but can be made statistically negligiblecited by 4
  2. 2025Approximation Bounds for Transformer Networks with Application to Regressioncited by 4
  3. 2025The Effect of Attention Head Count on Transformer Approximationcited by 4
  4. 2025Approximation of Permutation Invariant Polynomials by Transformers: Efficient Construction in Column-Sizecited by 3
  5. 2025Transformers for Learning on Noisy and Task-Level Manifolds: Approximation and Generalization Insightscited by 3
  6. 2024Universal Approximation of Operators with Transformers and Neural Integral Operatorscited by 2
  7. 2025A unified framework for establishing the universal approximation of transformer-type architecturescited by 2
  8. 2025A Mathematical Explanation of Transformerscited by 2
  9. 2026On the Expressive Power of Floating-Point Transformerscited by 2
  10. 2026Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizercited by 2
  11. 2025Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regressioncited by 1
  12. 2026On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functionscited by 1
  13. 2026Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unitycited by 1
  14. 2026Training-Free Universal Approximation by Prompting Random Transformerscited by 1
  15. 2025Overfitting has a limitation: a model-independent generalization gap bound based on R'enyi entropycited by 0
  16. 2025On the Universality of Transformer Architectures; How Much Attention Is Enough?cited by 0
  17. 2026Approximation Error Upper and Lower Bounds for H"older Class with Transformerscited by 0
  18. 2026Provable Data Scaling Law for Meta Learning via Complexity Minimizationcited by 0
  19. 2026Vector Search As Nearest Neighbor Matching: RAG-based Policy Learning in Causal Inferencecited by 0
  20. 2026Universal Approximation of Measure-to-Measure Operators by Pushforwardscited by 0