mlsanityData to 14 Oct 2026

A multiscale analysis of mean-field transformers in the moderate interaction regime

arXiv 2509.25040 ↗First posted 29 Sep 2025cs.LG math.PR stat.MLcited by 25

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the evolution of tokens through the depth of encoder-only transformer models at inference time by modeling them as a system of particles interacting in a mean-field way and studying the corresponding dynamics. More specifically, we consider this problem in the moderate interaction regime, where the number N of tokens is large and the inverse temperature parameter β of the model scales together with N. In this regime, the dynamics of the system displays a multiscale behavior: a fast phase, where the token empirical measure collapses on a low-dimensional space, an intermediate phase, where the measure further collapses into clusters, and a slow one, where such clusters sequentially merge into a single one. We provide a rigorous characterization of the limiting dynamics in each of these phases and prove convergence in the above mentioned limit, exemplifying our results with some simulations.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 27 of its 49 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

27 of 49 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2018Neural Ordinary Differential Equationscited by 3,779
  4. 2020Reformer: The Efficient Transformercited by 1,668
  5. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  8. 2023A mathematical perspective on Transformerscited by 164
  9. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  10. 2023The emergence of clusters in self-attention dynamicscited by 140
  11. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
  12. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 106
15 more
  1. 2023Learning time-scales in two-layers neural networkscited by 57
  2. 2024Dynamic metastability in the self-attention modelcited by 46
  3. 2025Scalable-Softmax Is Superior for Attentioncited by 45
  4. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 39
  5. 2024Clustering in Causal Attention Maskingcited by 32
  6. 2025A Unified Perspective on the Dynamics of Deep Transformerscited by 32
  7. 2024Measure-to-measure interpolation using Transformerscited by 29
  8. 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 26
  9. 2020Quantitative Propagation of Chaos for SGD in Wide Neural Networkscited by 20
  10. 2020Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regimecited by 17
  11. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17
  12. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 17
  13. 2024Clustering in pure-attention hardmax transformers and its role in sentiment analysiscited by 15
  14. 2025Synchronization of mean-field models on the circlecited by 12
  15. 2025Attention's forward pass and Frank-Wolfecited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 21.

25 papers.

Sort:

  1. 2025The Mean-Field Dynamics of Transformerscited by 30
  2. 2025Critical attention scaling in long-context transformerscited by 20
  3. 2025On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformerscited by 12
  4. 2025Normalization in Attention Dynamicscited by 9
  5. 2026Perceptrons and localization of attention's mean-field landscapecited by 8
  6. 2026Krause Synchronization Transformerscited by 5
  7. 2026Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regimecited by 5
  8. 2026Random Quadratic Form on a Sphere: Synchronization by Common Noisecited by 4
  9. 2026Scaling Limits of Long-Context Transformerscited by 3
  10. 2026A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phasescited by 3
  11. 2026Propagation of Chaos in Contextual Flow Mapscited by 3
  12. 2026On the Diverse Dynamical Behaviors Arising in Deep Linear Transformerscited by 3
13 more
  1. 2026YuriiFormer: A Suite of Nesterov-Accelerated Transformerscited by 2
  2. 2026Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformerscited by 1
  3. 2026L2 over Wasserstein: Statistical Analysis for Optimal Transportcited by 1
  4. 2026Reachability and asymptotics of Gaussian Transformer dynamicscited by 1
  5. 2026Clustered Attractor Manifolds and Dynamical Condensation in Self-Attentioncited by 1
  6. 2026Explanation of Dynamic Physical Field Predictions using WassersteinGrad: Application to Autoregressive Weather Forecastingcited by 0
  7. 2026Kinetic theory for Transformers and the lost-in-the-middle phenomenoncited by 0
  8. 2026A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attentioncited by 0
  9. 2026Random Quadratic Form with random forcing: Metastable synchronization by noisecited by 0
  10. 2026Opinion Leader Dynamics: How Sparse Attention Shapes Token Clusteringcited by 0
  11. 2026Sharp Convergence of Wasserstein Gradient Flows for Spectrally Nonnegative Interaction Energiescited by 0
  12. 2026The Role of Feed-Forward Layers in Transformer Dynamicscited by 0
  13. 2026Pattern Formation in Transformerscited by 0