mlsanityData to 7 Oct 2026

On the rate of convergence of a classifier based on a Transformer encoder

arXiv 2111.14574 ↗First posted 29 Nov 2021math.ST cs.LG stat.THcited by 16

Not on the map. Why?

Abstract (arXiv)

Pattern recognition based on a high-dimensional predictor is considered. A classifier is defined which is based on a Transformer encoder. The rate of convergence of the misclassification probability of the classifier towards the optimal misclassification probability is analyzed. It is shown that this classifier is able to circumvent the curse of dimensionality provided the aposteriori probability satisfies a suitable hierarchical composition model. Furthermore, the difference between Transformer classifiers analyzed theoretically in this paper and Transformer classifiers used nowadays in practice are illustrated by considering classification problems in natural language processing.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 17 of its 30 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

17 of 30 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2016Error bounds for approximations with deep ReLU networkscited by 567
  5. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  6. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  7. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  8. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 262
  9. 2020Deep Network Approximation for Smooth Functionscited by 163
  10. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  11. 2019The phase diagram of approximation rates for deep neural networkscited by 86
  12. 2019Deep ReLU network approximation of functions on a manifoldcited by 73
5 more
  1. 2018Deep Neural Networks Learn Non-Smooth Functions Effectivelycited by 57
  2. 2019Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov spacecited by 46
  3. 2021Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionalitycited by 34
  4. 2021Universal Approximation Under Constraints is Possible with Transformerscited by 32
  5. 2020Approximating smooth functions by deep neural networks with sigmoid activation functioncited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 1; 2025: 7; 2026: 2.

16 papers.

Sort:

  1. 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 46
  2. 2023Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Inputcited by 32
  3. 2023Approximation Rate of the Transformer Architecture for Sequence Modelingcited by 28
  4. 2025Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methodscited by 13
  5. 2024On the Optimal Memorization Capacity of Transformerscited by 10
  6. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5
  7. 2022Attention Enables Zero Approximation Errorcited by 4
  8. 2025Hallucinations are inevitable but can be made statistically negligiblecited by 4
  9. 2025Approximation Bounds for Transformer Networks with Application to Regressioncited by 4
  10. 2025Transformers for Learning on Noisy and Task-Level Manifolds: Approximation and Generalization Insightscited by 3
  11. 2023Transformers are Universal Predictorscited by 2
  12. 2023On the rate of convergence of an over-parametrized Transformer classifier learned by gradient descentcited by 2
4 more
  1. 2025CHEM: Estimating and Understanding Hallucinations in Deep Learning for Image Processingcited by 1
  2. 2026Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unitycited by 1
  3. 2025Overfitting has a limitation: a model-independent generalization gap bound based on R'enyi entropycited by 0
  4. 2026Approximation Error Upper and Lower Bounds for H"older Class with Transformerscited by 0