mlsanityData to 7 Oct 2026

A mathematical perspective on Transformers

arXiv 2312.10794 ↗First posted 17 Dec 2023cs.LG math.AP math.DScited by 160

Not on the map. Why?

Abstract (arXiv)

Transformers play a central role in the inner workings of large language models. We develop a mathematical framework for analyzing Transformers based on their interpretation as interacting particle systems, which reveals that clusters emerge in long time. Our study explores the underlying theory and offers new perspectives for mathematicians as well as computer scientists.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Builds on 40 of its 107 references that are in the map. First posted on arXiv 17 Dec 2023.

Builds on

40 of 107 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  8. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  9. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  10. 2021A Survey of Transformerscited by 399
  11. 2017Stable Architectures for Deep Neural Networkscited by 340
  12. 2021DeepViT: Towards Deeper Vision Transformercited by 288
28 more
  1. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  2. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  3. 2023The emergence of clusters in self-attention dynamicscited by 133
  4. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  5. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  6. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  7. 2017Maximum Principle Based Algorithms for Deep Learningcited by 77
  8. 2019Approximation Capabilities of Neural ODEs and Invertible Residual Networkscited by 56
  9. 2021Neural ODE control for classification, approximation and transportcited by 56
  10. 2020A Mathematical Theory of Attentioncited by 43
  11. 2023Sumformer: Universal Approximation for Efficient Transformerscited by 30
  12. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 29
  13. 2019Deep neural networks, generic universal interpolation, and controlled ODEscited by 28
  14. 2023Approximation Rate of the Transformer Architecture for Sequence Modelingcited by 28
  15. 2019A consensus-based global optimization method for high dimensional machine learning problemscited by 23
  16. 2016A consensus-based model for global optimization and its mean-field limitcited by 21
  17. 2020A Functional Perspective on Learning Symmetric Functions with Neural Networkscited by 16
  18. 2023A Brief Survey on the Approximation Theory for Sequence Modellingcited by 15
  19. 2023Interpolation, Approximation and Controllability of Deep Neural Networkscited by 15
  20. 2023Deep learning probability flows and entropy production rates in active mattercited by 15
  21. 2018Stochastic Deep Networkscited by 13
  22. 2020Consensus-Based Optimization on the Sphere: Convergence to Global Minimizers and Machine Learningcited by 12
  23. 2018On the Landscape of Synchronization Networks: A Perspective from Nonconvex Optimizationcited by 11
  24. 2022Turnpike in optimal control of PDEs, ResNets, and beyondcited by 10
  25. 2023Introduction to Transformers: an NLP Perspectivecited by 10
  26. 2022Expander graphs are globally synchronizingcited by 8
  27. 2021A Class of Dimension-free Metrics for the Convergence of Empirical Measurescited by 5
  28. 2022Nonlinear controllability and function representation by neural stochastic differential equationscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 29; 2025: 60; 2026: 70.

160 papers.

Sort:

By area:

  1. 2025Why do LLMs attend to the first token?cited by 122
  2. 2024A Survey on LoRA of Large Language Modelscited by 67
  3. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47
  4. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
  5. 2023How Smooth Is Attention?cited by 44
  6. 2024Transformers are Universal In-context Learnerscited by 42
  7. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 35
  8. 2024Clustering in Causal Attention Maskingcited by 31
  9. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  10. 2024Measure-to-measure interpolation using Transformerscited by 28
  11. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  12. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 24
38 more
  1. 2025The Mean-Field Dynamics of Transformerscited by 24
  2. 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 23
  3. 2024Provably learning a multi-head attention layercited by 21
  4. 2024Learning diffusion at lightspeedcited by 21
  5. 2024SINDER: Repairing the Singular Defects of DINOv2cited by 18
  6. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17
  7. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  8. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  9. 2025Critical attention scaling in long-context transformerscited by 17
  10. 2025Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddingscited by 16
  11. 2024Clustering in pure-attention hardmax transformers and its role in sentiment analysiscited by 15
  12. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 15
  13. 2024Implicit Optimization Bias of Next-Token Prediction in Linear Modelscited by 14
  14. 2025Token embeddings violate the manifold hypothesiscited by 14
  15. 2025Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methodscited by 13
  16. 2024The Asymptotic Behavior of Attention in Transformerscited by 12
  17. 2025Learning Enhanced Ensemble Filterscited by 11
  18. 2025Synchronization of mean-field models on the circlecited by 10
  19. 2025On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformerscited by 10
  20. 2024Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformerscited by 9
  21. 2025Normalization in Attention Dynamicscited by 9
  22. 2025Hierarchical Refinement: Optimal Transport to Infinity and Beyondcited by 8
  23. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  24. 2024Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamicscited by 7
  25. 2024Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysiscited by 7
  26. 2024Universal Approximation of Mean-Field Models via Transformerscited by 7
  27. 2024Low-Rank Optimal Transport through Factor Relaxation with Latent Couplingcited by 7
  28. 2025OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularizationcited by 7
  29. 2025Optimizing Rank for High-Fidelity Implicit Neural Representationscited by 7
  30. 2026Perceptrons and localization of attention's mean-field landscapecited by 7
  31. 2025Multistability of Self-Attention Dynamics in Transformerscited by 6
  32. 2025Attention-Only Transformers via Unrolled Subspace Denoisingcited by 5
  33. 2025Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physicscited by 5
  34. 2025Transformers through the lens of support-preserving maps between measurescited by 5
  35. 2025Optimal and Diffusion Transports in Machine Learningcited by 5
  36. 2026Clustering in Deep Stochastic Transformerscited by 5
  37. 2026Latent Semantic Manifolds in Large Language Modelscited by 5
  38. 2026Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topologycited by 5

50 most cited of 160 All 160 (JSON)