mlsanityData to 7 Oct 2026

Quantitative Clustering in Mean-Field Transformer Models

arXiv 2504.14697 ↗First posted 20 Apr 2025cs.LG math.AP math.DScited by 23

Not on the map. Why?

Abstract (arXiv)

The evolution of tokens through deep transformer models can be modeled as an interacting particle system that has been shown to exhibit an asymptotic clustering behavior akin to the synchronization phenomenon in Kuramoto models. In this work, we investigate the long-time clustering of mean-field transformer models. More precisely, under suitable assumptions on the transformer model parameters, we establish that any suitably regular mean-field initialization synchronizes exponentially fast to a Dirac point mass, with explicit quantitative convergence rates.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 17 of its 46 references that are in the map. First posted on arXiv 20 Apr 2025.

Builds on

17 of 46 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2023A mathematical perspective on Transformerscited by 160
  6. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  7. 2023The emergence of clusters in self-attention dynamicscited by 133
  8. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  9. 2024Convergence Analysis of Probability Flow ODE for Score-based Generative Modelscited by 56
  10. 2024Statistical optimal transportcited by 44
  11. 2024Dynamic metastability in the self-attention modelcited by 42
  12. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 35
5 more
  1. 2024Clustering in Causal Attention Maskingcited by 31
  2. 2025A Unified Perspective on the Dynamics of Deep Transformerscited by 30
  3. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17
  4. 2024The Asymptotic Behavior of Attention in Transformerscited by 12
  5. 2022Expander graphs are globally synchronizingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 15.

23 papers.

Sort:

  1. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 24
  2. 2025Critical attention scaling in long-context transformerscited by 17
  3. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 15
  4. 2025Synchronization of mean-field models on the circlecited by 10
  5. 2025On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformerscited by 10
  6. 2025Normalization in Attention Dynamicscited by 9
  7. 2026Perceptrons and localization of attention's mean-field landscapecited by 7
  8. 2026Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regimecited by 4
  9. 2025Token Sample Complexity of Attentioncited by 3
  10. 2026Krause Synchronization Transformerscited by 3
  11. 2026YuriiFormer: A Suite of Nesterov-Accelerated Transformerscited by 2
  12. 2026Random Quadratic Form on a Sphere: Synchronization by Common Noisecited by 2
11 more
  1. 2026Propagation of Chaos in Contextual Flow Mapscited by 2
  2. 2026Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attentioncited by 1
  3. 2026Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformerscited by 1
  4. 2026Scaling Limits of Long-Context Transformerscited by 1
  5. 2026Measure-to-measure Regression with Transformerscited by 1
  6. 2025TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformationscited by 0
  7. 2026Kinetic theory for Transformers and the lost-in-the-middle phenomenoncited by 0
  8. 2026Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flowscited by 0
  9. 2026Random Quadratic Form with random forcing: Metastable synchronization by noisecited by 0
  10. 2026Opinion Leader Dynamics: How Sparse Attention Shapes Token Clusteringcited by 0
  11. 2026Sharp Convergence of Wasserstein Gradient Flows for Spectrally Nonnegative Interaction Energiescited by 0