mlsanityData to 7 Oct 2026

Global Convergence in Training Large-Scale Transformers

arXiv 2410.23610 ↗First posted 31 Oct 2024stat.ML cs.LG math.STcited by 5

Not on the map. Why?

Abstract (arXiv)

Despite the widespread success of Transformers across various domains, their optimization guarantees in large-scale model settings are not well-understood. This paper rigorously analyzes the convergence properties of gradient flow in training Transformers with weight decay regularization. First, we construct the mean-field limit of large-scale Transformers, showing that as the model width and depth go to infinity, gradient flow converges to the Wasserstein gradient flow, which is represented by a partial differential equation. Then, we demonstrate that the gradient flow reaches a global minimum consistent with the PDE solution when the weight decay regularization parameter is sufficiently small. Our analysis is based on a series of novel mean-field techniques that adapt to Transformers. Compared with existing tools for deep networks (Lu et al., 2020) that demand homogeneity and global Lipschitz smoothness, we utilize a refined analysis assuming only textitpartial homogeneity and textitlocal Lipschitz smoothness. These new techniques may be of independent interest.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 42 of its 76 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

42 of 76 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  10. 2018On Lazy Training in Differentiable Programmingcited by 671
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  12. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
30 more
  1. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  2. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  3. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  4. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  5. 2020The Lipschitz Constant of Self-Attentioncited by 171
  6. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  7. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  8. 2023In-Context Convergence of Transformerscited by 112
  9. 2022Vision Transformers provably learn spatial structurecited by 92
  10. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  11. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  12. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  13. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  14. 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 46
  15. 2020A Mathematical Theory of Attentioncited by 43
  16. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  17. 2023Memorization Capacity of Multi-Head Attention in Transformerscited by 39
  18. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  19. 2021Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networkscited by 38
  20. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 38
  21. 2023Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Inputcited by 32
  22. 2018An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networkscited by 25
  23. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  24. 2023Interpolation, Approximation and Controllability of Deep Neural Networkscited by 15
  25. 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 10
  26. 2021Global convergence of ResNets: From finite to infinite width using linear parameterizationcited by 9
  27. 2016Residual Networks Behave Like Ensembles of Relatively Shallow Networkscited by 8
  28. 2019Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representationscited by 7
  29. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  30. 2023Alternatives to the Scaled Dot Product for Attention in the Transformer Neural Network Architecturecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026Convergent Stochastic Training of Multi-Headed Attention and Understanding LoRAcited by 1
  2. 2026Training Infinitely Deep and Wide Transformerscited by 1
  3. 2025Internal Planning in Language Models: Characterizing Horizon and Branch Awarenesscited by 0
  4. 2025Transformers for Tabular Data: A Training Perspective of Self-Attention via Optimal Transportcited by 0
  5. 2026Man, Machine, and Mathematicscited by 0