mlsanityData to 14 Oct 2026

On the Convergence of Encoder-only Shallow Transformers

arXiv 2311.01575 ↗First posted 2 Nov 2023cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper, we aim to build the global convergence theory of encoder-only shallow Transformers under a realistic setting from the perspective of architectures, initialization, and scaling under a finite width regime. The difficulty lies in how to tackle the softmax in self-attention mechanism, the core ingredient of Transformer. In particular, we diagnose the scaling scheme, carefully tackle the input/output of softmax, and prove that quadratic overparameterization is sufficient for global convergence of our shallow Transformers under commonly-used He/LeCun initialization in practice. Besides, neural tangent kernel (NTK) based analysis is also given, which facilitates a comprehensive comparison. Our theory demonstrates the separation on the importance of different scaling schemes and initialization. We believe our results can pave the way for a better understanding of modern Transformers, particularly on training dynamics.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 49 of its 61 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

49 of 61 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2017Non-local Neural Networkscited by 2,293
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  8. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  9. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  10. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  11. 2022Transformers learn in-context by gradient descentcited by 732
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
37 more
  1. 2018On Lazy Training in Differentiable Programmingcited by 675
  2. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  3. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  6. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  7. 2022How Do Vision Transformers Work?cited by 307
  8. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  9. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  10. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  11. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  12. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  13. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  14. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  15. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  16. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  17. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  18. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
  19. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 121
  20. 2022Vision Transformers provably learn spatial structurecited by 93
  21. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  22. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  23. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  24. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  25. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  26. 2020The Recurrent Neural Tangent Kernelcited by 67
  27. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  28. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  29. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  30. 2020Kernel regression in high dimensions: Refined analysis beyond double descentcited by 39
  31. 2021Approximating How Single Head Attention Learnscited by 38
  32. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  33. 2018On the Spectrum of Random Features Maps of High Dimensional Datacited by 28
  34. 2022Transformers from an Optimization Perspectivecited by 27
  35. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  36. 2020Kernel-Based Smoothness Analysis of Residual Networkscited by 19
  37. 2022Generalization Properties of NAS under Activation and Skip Connection Searchcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024Unraveling the Gradient Descent Dynamics of Transformerscited by 11
  2. 2024Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysiscited by 9
  3. 2024Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignmentcited by 9
  4. 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
  5. 2025On the Convergence of Gradient Descent on Learning Transformers with Residual Connectionscited by 6
  6. 2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
  7. 2026Learning to Adapt: In-Context Learning Beyond Stationaritycited by 2
  8. 2024Generalizable autoregressive modeling of time series through functional narrativescited by 1
  9. 2025Depth-induced NTK: Bridging Over-parameterized Neural Networks and Deep Neural Kernelscited by 1
  10. 2026Finite-Time Analysis of Gradient Descent for Shallow Transformerscited by 1
  11. 2026Training Infinitely Deep and Wide Transformerscited by 1
  12. 2025A Second-Order Perspective on Pruning at Initialization and Knowledge Transfercited by 0
1 more
  1. 2026Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidancecited by 0