# The Mean-Field Dynamics of Transformers

arXiv: https://arxiv.org/abs/2512.01868 · HTML: https://mlsanity.com/paper/2512.01868/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 67 references that are in the map.
- First posted on arXiv 1 Dec 2025.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse](https://mlsanity.com/paper/2206.03126/) (2022)
- [The emergence of clusters in self-attention dynamics](https://mlsanity.com/paper/2305.05465/) (2023)
- [Sinkformers: Transformers with Doubly Stochastic Attention](https://mlsanity.com/paper/2110.11773/) (2021)
- [Maximum Principle Based Algorithms for Deep Learning](https://mlsanity.com/paper/1710.09513/) (2017)
- [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://mlsanity.com/paper/2410.01131/) (2024)
- [A Mean-Field Optimal Control Formulation of Deep Learning](https://mlsanity.com/paper/1807.01083/) (2018)
- [Neural ODE control for classification, approximation and transport](https://mlsanity.com/paper/2104.05278/) (2021)
- [Dynamic metastability in the self-attention model](https://mlsanity.com/paper/2410.06833/) (2024)
- [Scalable-Softmax Is Superior for Attention](https://mlsanity.com/paper/2501.19399/) (2025)
- [Emergence of meta-stable clustering in mean-field transformer models](https://mlsanity.com/paper/2410.23228/) (2024)
- [Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN](https://mlsanity.com/paper/2412.13795/) (2024)
- [Clustering in Causal Attention Masking](https://mlsanity.com/paper/2411.04990/) (2024)
- [A Unified Perspective on the Dynamics of Deep Transformers](https://mlsanity.com/paper/2501.18322/) (2025)
- [A multiscale analysis of mean-field transformers in the moderate interaction regime](https://mlsanity.com/paper/2509.25040/) (2025)
- [Geometric Dynamics of Signal Propagation Predict Trainability of Transformers](https://mlsanity.com/paper/2403.02579/) (2024)
- [Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks](https://mlsanity.com/paper/2402.09092/) (2024)

## Built on by (most cited first, 30 of 30)

- [Krause Synchronization Transformers](https://mlsanity.com/paper/2602.11534/) (2026)
- [Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime](https://mlsanity.com/paper/2605.10931/) (2026)
- [Random Quadratic Form on a Sphere: Synchronization by Common Noise](https://mlsanity.com/paper/2603.06187/) (2026)
- [Propagation of Chaos in Contextual Flow Maps](https://mlsanity.com/paper/2605.16747/) (2026)
- [On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers](https://mlsanity.com/paper/2607.18584/) (2026)
- [YuriiFormer: A Suite of Nesterov-Accelerated Transformers](https://mlsanity.com/paper/2601.23236/) (2026)
- [Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention](https://mlsanity.com/paper/2605.04279/) (2026)
- [The physics of AI weather models](https://mlsanity.com/paper/2605.23778/) (2026)
- [Physics-based phenomenological characterization of cross-modal bias in multimodal models](https://mlsanity.com/paper/2602.20624/) (2026)
- [Static and Dynamic Approaches to Computing Barycenters of Probability Measures on Graphs](https://mlsanity.com/paper/2603.26940/) (2026)
- [Diffusion Operator Geometry of Feedforward Representations](https://mlsanity.com/paper/2605.01107/) (2026)
- [Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers](https://mlsanity.com/paper/2605.07772/) (2026)
- [L2 over Wasserstein: Statistical Analysis for Optimal Transport](https://mlsanity.com/paper/2605.21365/) (2026)
- [Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention](https://mlsanity.com/paper/2608.08922/) (2026)
- [Universal Approximation of Measure-to-Measure Operators by Pushforwards](https://mlsanity.com/paper/2609.35483/) (2026)
- [Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classification](https://mlsanity.com/paper/2604.11613/) (2026)
- [Explanation of Dynamic Physical Field Predictions using WassersteinGrad: Application to Autoregressive Weather Forecasting](https://mlsanity.com/paper/2604.22580/) (2026)
- [Uniform Scaling Limits in AdamW-Trained Transformers](https://mlsanity.com/paper/2605.11059/) (2026)
- [A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention](https://mlsanity.com/paper/2605.12697/) (2026)
- [Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics](https://mlsanity.com/paper/2605.29351/) (2026)
- [Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables](https://mlsanity.com/paper/2605.30229/) (2026)
- [Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere](https://mlsanity.com/paper/2607.24502/) (2026)
- [Many-body Tipping Dynamics of ChatGPT-like AIs](https://mlsanity.com/paper/2607.25279/) (2026)
- [Random Quadratic Form with random forcing: Metastable synchronization by noise](https://mlsanity.com/paper/2608.16664/) (2026)
- [Opinion Leader Dynamics: How Sparse Attention Shapes Token Clustering](https://mlsanity.com/paper/2609.24202/) (2026)
- [Sharp Convergence of Wasserstein Gradient Flows for Spectrally Nonnegative Interaction Energies](https://mlsanity.com/paper/2609.27008/) (2026)
- [Relative Generalization Invariance of LLM Pretraining](https://mlsanity.com/paper/2609.33016/) (2026)
- [Understanding Clustering in Slot Attention via Particle Dynamics](https://mlsanity.com/paper/2610.04493/) (2026)
- [Deep learning from a control vantage point](https://mlsanity.com/paper/2610.14385/) (2026)
- [Asymptotic Behavior of Transformers](https://mlsanity.com/paper/2610.19358/) (2026)

## Abstract (arXiv)

We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein gradient flows, synchronization models (Kuramoto), and mean-shift clustering. Central to our results is a global clustering phenomenon whereby tokens cluster asymptotically after long metastable states where they are arranged into multiple clusters. We further analyze a tractable equiangular reduction to obtain exact clustering rates, show how commonly used normalization schemes alter contraction speeds, and identify a phase transition for long-context attention. The results highlight both the mechanisms that drive representation collapse and the regimes that preserve expressive, multi-cluster structure in deep attention architectures.