mlsanityData to 7 Oct 2026

The Mean-Field Dynamics of Transformers

arXiv 2512.01868 ↗First posted 1 Dec 2025cs.LG math-ph math.DScited by 24

Not on the map. Why?

Abstract (arXiv)

We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein gradient flows, synchronization models (Kuramoto), and mean-shift clustering. Central to our results is a global clustering phenomenon whereby tokens cluster asymptotically after long metastable states where they are arranged into multiple clusters. We further analyze a tractable equiangular reduction to obtain exact clustering rates, show how commonly used normalization schemes alter contraction speeds, and identify a phase transition for long-context attention. The results highlight both the mechanisms that drive representation collapse and the regimes that preserve expressive, multi-cluster structure in deep attention architectures.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 41 of its 67 references that are in the map. First posted on arXiv 1 Dec 2025.

Builds on

41 of 67 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2018Neural Ordinary Differential Equationscited by 3,733
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
29 more
  1. 2017Stable Architectures for Deep Neural Networkscited by 340
  2. 2023A mathematical perspective on Transformerscited by 160
  3. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  4. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  5. 2023The emergence of clusters in self-attention dynamicscited by 133
  6. 2017Maximum Principle Based Algorithms for Deep Learningcited by 77
  7. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  8. 2018A Mean-Field Optimal Control Formulation of Deep Learningcited by 67
  9. 2021Neural ODE control for classification, approximation and transportcited by 56
  10. 2024Dynamic metastability in the self-attention modelcited by 42
  11. 2025Scalable-Softmax Is Superior for Attentioncited by 38
  12. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  13. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 35
  14. 2024Clustering in Causal Attention Maskingcited by 31
  15. 2025A Unified Perspective on the Dynamics of Deep Transformerscited by 30
  16. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  17. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 24
  18. 2024Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networkscited by 21
  19. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17
  20. 2025Critical attention scaling in long-context transformerscited by 17
  21. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 16
  22. 2024Clustering in pure-attention hardmax transformers and its role in sentiment analysiscited by 15
  23. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 15
  24. 2025Synchronization of mean-field models on the circlecited by 10
  25. 2025On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformerscited by 10
  26. 2025The power of fine-grained experts: Granularity boosts expressivity in Mixture of Expertscited by 9
  27. 2025Normalization in Attention Dynamicscited by 9
  28. 2015Consensus Convergence with Stochastic Effectscited by 4
  29. 2024On the number of modes of Gaussian kernel density estimatorscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 24.

24 papers.

Sort:

  1. 2026Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regimecited by 4
  2. 2026Krause Synchronization Transformerscited by 3
  3. 2026YuriiFormer: A Suite of Nesterov-Accelerated Transformerscited by 2
  4. 2026Random Quadratic Form on a Sphere: Synchronization by Common Noisecited by 2
  5. 2026Propagation of Chaos in Contextual Flow Mapscited by 2
  6. 2026Physics-based phenomenological characterization of cross-modal bias in multimodal modelscited by 1
  7. 2026Diffusion Operator Geometry of Feedforward Representationscited by 1
  8. 2026Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attentioncited by 1
  9. 2026Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformerscited by 1
  10. 2026L2 over Wasserstein: Statistical Analysis for Optimal Transportcited by 1
  11. 2026The physics of AI weather modelscited by 1
  12. 2026Static and Dynamic Approaches to Computing Barycenters of Probability Measures on Graphscited by 0
12 more
  1. 2026Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classificationcited by 0
  2. 2026Explanation of Dynamic Physical Field Predictions using WassersteinGrad: Application to Autoregressive Weather Forecastingcited by 0
  3. 2026A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attentioncited by 0
  4. 2026Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variablescited by 0
  5. 2026Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Spherecited by 0
  6. 2026Many-body Tipping Dynamics of ChatGPT-like AIscited by 0
  7. 2026Clustered Attractor Manifolds and Dynamical Condensation in Self-Attentioncited by 0
  8. 2026Random Quadratic Form with random forcing: Metastable synchronization by noisecited by 0
  9. 2026Opinion Leader Dynamics: How Sparse Attention Shapes Token Clusteringcited by 0
  10. 2026Sharp Convergence of Wasserstein Gradient Flows for Spectrally Nonnegative Interaction Energiescited by 0
  11. 2026Relative Generalization Invariance of LLM Pretrainingcited by 0
  12. 2026Universal Approximation of Measure-to-Measure Operators by Pushforwardscited by 0