mlsanityData to 7 Oct 2026

The emergence of clusters in self-attention dynamics

arXiv 2305.05465 ↗First posted 9 May 2023cs.LG math.AP stat.MLcited by 133

Not on the map. Why?

Abstract (arXiv)

Viewing Transformers as interacting particle systems, we describe the geometry of learned representations when the weights are not time dependent. We show that particles, representing tokens, tend to cluster toward particular limiting objects as time tends to infinity. Cluster locations are determined by the initial tokens, confirming context-awareness of representations learned by Transformers. Using techniques from dynamical systems and partial differential equations, we show that the type of limiting object that emerges depends on the spectrum of the value matrix. Additionally, in the one-dimensional case we prove that the self-attention matrix converges to a low-rank Boolean matrix. The combination of these results mathematically confirms the empirical observation made by Vaswani et al. [VSP'17] that leaders appear in a sequence of tokens when processed by Transformers.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Builds on 18 of its 41 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

18 of 41 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  9. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  10. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  11. 2021A Survey of Transformerscited by 399
  12. 2017Stable Architectures for Deep Neural Networkscited by 340
6 more
  1. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  2. 2020Fast Transformers with Clustered Attentioncited by 123
  3. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  4. 2018A Mean-Field Optimal Control Formulation of Deep Learningcited by 67
  5. 2013On properties of the Generalized Wasserstein distancecited by 11
  6. 2014Control to flocking of the kinetic Cucker-Smale modelcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 30; 2025: 43; 2026: 56.

133 papers.

Sort:

By area:

  1. 2023A mathematical perspective on Transformerscited by 160
  2. 2024A Survey on LoRA of Large Language Modelscited by 67
  3. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
  4. 2023How Smooth Is Attention?cited by 44
  5. 2024Transformers are Universal In-context Learnerscited by 42
  6. 2024Continuum Attention for Neural Operatorscited by 39
  7. 2024Artificial Kuramoto Oscillatory Neuronscited by 37
  8. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 35
  9. 2024Clustering in Causal Attention Maskingcited by 31
  10. 2024Measure-to-measure interpolation using Transformerscited by 28
  11. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  12. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 24
38 more
  1. 2025The Mean-Field Dynamics of Transformerscited by 24
  2. 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 23
  3. 2024Provably learning a multi-head attention layercited by 21
  4. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
  5. 2024SINDER: Repairing the Singular Defects of DINOv2cited by 18
  6. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  7. 2025Critical attention scaling in long-context transformerscited by 17
  8. 2025Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddingscited by 16
  9. 2025Block-Recurrent Dynamics in Vision Transformerscited by 16
  10. 2024Clustering in pure-attention hardmax transformers and its role in sentiment analysiscited by 15
  11. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 15
  12. 2026The Newton-Muon Optimizercited by 14
  13. 2024The Asymptotic Behavior of Attention in Transformerscited by 12
  14. 2024Residual Connections Harm Generative Representation Learningcited by 10
  15. 2025Synchronization of mean-field models on the circlecited by 10
  16. 2024Transformer Block Coupling and its Correlation with Generalization in LLMscited by 9
  17. 2024Towards Understanding the Universality of Transformers for Next-Token Predictioncited by 9
  18. 2025Normalization in Attention Dynamicscited by 9
  19. 2026Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoningcited by 9
  20. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  21. 2024Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamicscited by 7
  22. 2024Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysiscited by 7
  23. 2024Elliptical Attentioncited by 7
  24. 2025OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularizationcited by 7
  25. 2025Solving Empirical Bayes via Transformerscited by 7
  26. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  27. 2026Perceptrons and localization of attention's mean-field landscapecited by 7
  28. 2025Quantum Doubly Stochastic Transformerscited by 6
  29. 2024Bridging Associative Memory and Probabilistic Modelingcited by 5
  30. 2024Iterated Schr"odinger bridge approximation to Wasserstein Gradient Flowscited by 5
  31. 2025Attention-Only Transformers via Unrolled Subspace Denoisingcited by 5
  32. 2025Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physicscited by 5
  33. 2025Transformers through the lens of support-preserving maps between measurescited by 5
  34. 2025Optimal and Diffusion Transports in Machine Learningcited by 5
  35. 2026Clustering in Deep Stochastic Transformerscited by 5
  36. 2023Cluster-based classification with neural ODEs via controlcited by 4
  37. 2024An In-depth Investigation of Sparse Rate Reduction in Transformer-like Modelscited by 4
  38. 2024On the number of modes of Gaussian kernel density estimatorscited by 4

50 most cited of 133 All 133 (JSON)