mlsanityData to 7 Oct 2026

Emergence of meta-stable clustering in mean-field transformer models

arXiv 2410.23228 ↗First posted 30 Oct 2024cs.LG math.APcited by 35

Not on the map. Why?

Abstract (arXiv)

We model the evolution of tokens within a deep stack of Transformer layers as a continuous-time flow on the unit sphere, governed by a mean-field interacting particle system, building on the framework introduced in (Geshkovski et al., 2023). Studying the corresponding mean-field Partial Differential Equation (PDE), which can be interpreted as a Wasserstein gradient flow, in this paper we provide a mathematical investigation of the long-term behavior of this system, with a particular focus on the emergence and persistence of meta-stable phases and clustering phenomena, key elements in applications like next-token prediction. More specifically, we perform a perturbative analysis of the mean-field PDE around the iid uniform initialization and prove that, in the limit of large number of tokens, the model remains close to a meta-stable manifold of solutions with a given structure (e.g., periodicity). Further, the structure characterizing the meta-stable manifold is explicitly identified, as a function of the inverse temperature parameter of the model, by the index maximizing a certain rescaling of Gegenbauer polynomials.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 13 of its 36 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

13 of 36 references.

References · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  5. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  6. 2023A mathematical perspective on Transformerscited by 160
  7. 2023The emergence of clusters in self-attention dynamicscited by 133
  8. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  9. 2024Dynamic metastability in the self-attention modelcited by 42
  10. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  11. 2020Quantitative Propagation of Chaos for SGD in Wide Neural Networkscited by 20
  12. 2020Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regimecited by 17
1 more
  1. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 15; 2026: 17.

35 papers.

Sort:

  1. 2024Clustering in Causal Attention Maskingcited by 31
  2. 2024Measure-to-measure interpolation using Transformerscited by 28
  3. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 24
  4. 2025The Mean-Field Dynamics of Transformerscited by 24
  5. 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 23
  6. 2025Critical attention scaling in long-context transformerscited by 17
  7. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 15
  8. 2025Synchronization of mean-field models on the circlecited by 10
  9. 2025On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformerscited by 10
  10. 2025Normalization in Attention Dynamicscited by 9
  11. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  12. 2025OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularizationcited by 7
23 more
  1. 2026Perceptrons and localization of attention's mean-field landscapecited by 7
  2. 2026Clustering in Deep Stochastic Transformerscited by 5
  3. 2024On the number of modes of Gaussian kernel density estimatorscited by 4
  4. 2025Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobianscited by 4
  5. 2026Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regimecited by 4
  6. 2025Token Sample Complexity of Attentioncited by 3
  7. 2025Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspectivecited by 3
  8. 2026Krause Synchronization Transformerscited by 3
  9. 2026YuriiFormer: A Suite of Nesterov-Accelerated Transformerscited by 2
  10. 2026Random Quadratic Form on a Sphere: Synchronization by Common Noisecited by 2
  11. 2026Propagation of Chaos in Contextual Flow Mapscited by 2
  12. 2026Physics-based phenomenological characterization of cross-modal bias in multimodal modelscited by 1
  13. 2025Revisiting Transformers with Insights from Image Filtering and Boostingcited by 0
  14. 2025Minimax Rates for Learning Pairwise Interactions in Attention-Style Modelscited by 0
  15. 2026Kinetic theory for Transformers and the lost-in-the-middle phenomenoncited by 0
  16. 2026Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flowscited by 0
  17. 2026Reachability and asymptotics of Gaussian Transformer dynamicscited by 0
  18. 2026When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformerscited by 0
  19. 2026Clustered Attractor Manifolds and Dynamical Condensation in Self-Attentioncited by 0
  20. 2026Random Quadratic Form with random forcing: Metastable synchronization by noisecited by 0
  21. 2026Opinion Leader Dynamics: How Sparse Attention Shapes Token Clusteringcited by 0
  22. 2026Sharp Convergence of Wasserstein Gradient Flows for Spectrally Nonnegative Interaction Energiescited by 0
  23. 2026The Role of Feed-Forward Layers in Transformer Dynamicscited by 0