# Emergence of meta-stable clustering in mean-field transformer models

arXiv: https://arxiv.org/abs/2410.23228 · HTML: https://mlsanity.com/paper/2410.23228/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [The emergence of clusters in self-attention dynamics](https://mlsanity.com/paper/2305.05465/) (2023)
- [Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach](https://mlsanity.com/paper/1805.00915/) (2018)
- [Dynamic metastability in the self-attention model](https://mlsanity.com/paper/2410.06833/) (2024)
- [Geometric Dynamics of Signal Propagation Predict Trainability of Transformers](https://mlsanity.com/paper/2403.02579/) (2024)
- [Quantitative Propagation of Chaos for SGD in Wide Neural Networks](https://mlsanity.com/paper/2007.06352/) (2020)
- [Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime](https://mlsanity.com/paper/2010.11858/) (2020)
- [Synchronization on circles and spheres with nonlinear interactions](https://mlsanity.com/paper/2405.18273/) (2024)

## Built on by (most cited first, 30 of 39)

- [Clustering in Causal Attention Masking](https://mlsanity.com/paper/2411.04990/) (2024)
- [The Mean-Field Dynamics of Transformers](https://mlsanity.com/paper/2512.01868/) (2025)
- [Measure-to-measure interpolation using Transformers](https://mlsanity.com/paper/2411.04551/) (2024)
- [Quantitative Clustering in Mean-Field Transformer Models](https://mlsanity.com/paper/2504.14697/) (2025)
- [A multiscale analysis of mean-field transformers in the moderate interaction regime](https://mlsanity.com/paper/2509.25040/) (2025)
- [Critical attention scaling in long-context transformers](https://mlsanity.com/paper/2510.05554/) (2025)
- [Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation](https://mlsanity.com/paper/2505.24333/) (2025)
- [Synchronization of mean-field models on the circle](https://mlsanity.com/paper/2507.22857/) (2025)
- [On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformers](https://mlsanity.com/paper/2510.20094/) (2025)
- [Normalization in Attention Dynamics](https://mlsanity.com/paper/2510.22026/) (2025)
- [Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency](https://mlsanity.com/paper/2505.13499/) (2025)
- [Perceptrons and localization of attention's mean-field landscape](https://mlsanity.com/paper/2601.21366/) (2026)
- [Clustering in Deep Stochastic Transformers](https://mlsanity.com/paper/2601.21942/) (2026)
- [OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization](https://mlsanity.com/paper/2501.18793/) (2025)
- [Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians](https://mlsanity.com/paper/2505.19458/) (2025)
- [Krause Synchronization Transformers](https://mlsanity.com/paper/2602.11534/) (2026)
- [Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime](https://mlsanity.com/paper/2605.10931/) (2026)
- [On the number of modes of Gaussian kernel density estimators](https://mlsanity.com/paper/2412.09080/) (2024)
- [Random Quadratic Form on a Sphere: Synchronization by Common Noise](https://mlsanity.com/paper/2603.06187/) (2026)
- [Token Sample Complexity of Attention](https://mlsanity.com/paper/2512.10656/) (2025)
- [Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective](https://mlsanity.com/paper/2512.11784/) (2025)
- [Propagation of Chaos in Contextual Flow Maps](https://mlsanity.com/paper/2605.16747/) (2026)
- [On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers](https://mlsanity.com/paper/2607.18584/) (2026)
- [YuriiFormer: A Suite of Nesterov-Accelerated Transformers](https://mlsanity.com/paper/2601.23236/) (2026)
- [Physics-based phenomenological characterization of cross-modal bias in multimodal models](https://mlsanity.com/paper/2602.20624/) (2026)
- [Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows](https://mlsanity.com/paper/2605.18870/) (2026)
- [Reachability and asymptotics of Gaussian Transformer dynamics](https://mlsanity.com/paper/2606.07600/) (2026)
- [Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention](https://mlsanity.com/paper/2608.08922/) (2026)
- [Revisiting Transformers with Insights from Image Filtering and Boosting](https://mlsanity.com/paper/2506.10371/) (2025)
- [Minimax Rates for Learning Pairwise Interactions in Attention-Style Models](https://mlsanity.com/paper/2510.11789/) (2025)

## Abstract (arXiv)

We model the evolution of tokens within a deep stack of Transformer layers as a continuous-time flow on the unit sphere, governed by a mean-field interacting particle system, building on the framework introduced in (Geshkovski et al., 2023). Studying the corresponding mean-field Partial Differential Equation (PDE), which can be interpreted as a Wasserstein gradient flow, in this paper we provide a mathematical investigation of the long-term behavior of this system, with a particular focus on the emergence and persistence of meta-stable phases and clustering phenomena, key elements in applications like next-token prediction. More specifically, we perform a perturbative analysis of the mean-field PDE around the iid uniform initialization and prove that, in the limit of large number of tokens, the model remains close to a meta-stable manifold of solutions with a given structure (e.g., periodicity). Further, the structure characterizing the meta-stable manifold is explicitly identified, as a function of the inverse temperature parameter of the model, by the index maximizing a certain rescaling of Gegenbauer polynomials.