mlsanityData to 14 Oct 2026

Infinite Limits of Multi-head Transformer Dynamics

arXiv 2405.15712 ↗First posted 24 May 2024stat.ML cond-mat.dis-nn cs.LGcited by 39

Not on the map. Why?

Abstract (arXiv)

In this work, we analyze various scaling limits of the training dynamics of transformer models in the feature learning regime. We identify the set of parameterizations that admit well-defined infinite width and depth limits, allowing the attention layers to update throughout training--a relevant notion of feature learning in these models. We then use tools from dynamical mean field theory (DMFT) to analyze various infinite limits (infinite key/query dimension, infinite heads, and infinite depth) which have different statistical descriptions depending on which infinite limit is taken and how attention layers are scaled. We provide numerical evidence of convergence to the limits and discuss how the parameterization qualitatively influences learned features.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 38 of its 48 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

38 of 48 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  10. 2018On Lazy Training in Differentiable Programmingcited by 675
  11. 2020A Survey on Visual Transformercited by 654
  12. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
26 more
  1. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  2. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  3. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  4. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  5. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 121
  6. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  7. 2023A Spectral Condition for Feature Learningcited by 109
  8. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  9. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 89
  10. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  11. 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
  12. 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 57
  13. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  14. 2020On the distance between two neural networks and the stability of learningcited by 46
  15. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 43
  16. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  17. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 41
  18. 2023Simplifying Transformer Blockscited by 37
  19. 2019Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Modelscited by 28
  20. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  21. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  22. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  23. 2023Effective Theory of Transformers at Initializationcited by 21
  24. 2022The Influence of Learning Rule on Representation Dynamics in Wide Neural Networkscited by 19
  25. 2023Automatic Gradient Descent: Deep Learning without Hyperparameterscited by 14
  26. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 17; 2026: 16.

39 papers.

Sort:

By area:

  1. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  2. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  3. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
  4. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  5. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  6. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
  7. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 21
  8. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 17
  9. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 16
  10. 2024Progressive distillation induces an implicit curriculumcited by 15
  11. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  12. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 13
27 more
  1. 2024High-dimensional learning of narrow neural networkscited by 12
  2. 2025Scaling Diffusion Transformers Efficiently via μPcited by 6
  3. 2025On the Convergence of Gradient Descent on Learning Transformers with Residual Connectionscited by 6
  4. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
  5. 2026Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfercited by 6
  6. 2024Global Convergence in Training Large-Scale Transformerscited by 5
  7. 2025Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programscited by 5
  8. 2025Function-Space Learning Ratescited by 4
  9. 2025Bayes optimal learning of attention-indexed modelscited by 3
  10. 2026Learning Rate Transfer in Normalized Transformerscited by 3
  11. 2025Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theorycited by 2
  12. 2026On the Infinite Width and Depth Limits of Predictive Coding Networkscited by 2
  13. 2026SafeRoPE: Risk-specific Head-wise Embedding Rotation for Safe Generation in Rectified Flow Transformerscited by 2
  14. 2026Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Ratecited by 2
  15. 2026Statistical Properties of Training & Generalizationcited by 2
  16. 2025Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrixcited by 1
  17. 2026Finite-Time Analysis of Gradient Descent for Shallow Transformerscited by 1
  18. 2026On the global convergence of gradient flow for wide shallow models beyond homogeneous nonlinearitiescited by 1
  19. 2026Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Modelscited by 1
  20. 2025Deep Neural Nets as Hamiltonianscited by 0
  21. 2025Revisiting Transformers with Insights from Image Filtering and Boostingcited by 0
  22. 2025Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimizationcited by 0
  23. 2026Neural Network Quantum Field Theory from Transformer Architecturescited by 0
  24. 2026A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attentioncited by 0
  25. 2026Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networkscited by 0
  26. 2026Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNetscited by 0
  27. 2026Learning reshapes power-law anisotropy in internal representationscited by 0