mlsanityData to 14 Oct 2026

Perceptrons and localization of attention's mean-field landscape

arXiv 2601.21366 ↗First posted 29 Jan 2026cs.LG math.OCcited by 8

Not on the map. Why?

Abstract (arXiv)

The forward pass of a Transformer can be seen as an interacting particle system on the unit sphere: time plays the role of layers, particles that of token embeddings, and the unit sphere idealizes layer normalization. In some weight settings the system can even be seen as a gradient flow for an explicit energy, and one can make sense of the infinite context length (mean-field) limit thanks to Wasserstein gradient flows. In this paper we study the effect of the perceptron block in this setting, and show that critical points are generically atomic and localized on subsets of the sphere.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

28 of 50 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018Neural Ordinary Differential Equationscited by 3,779
  3. 20242 OLMo 2 Furiouscited by 340
  4. 2023A mathematical perspective on Transformerscited by 164
  5. 2023The emergence of clusters in self-attention dynamicscited by 140
  6. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
  7. 2025Why do LLMs attend to the first token?cited by 127
  8. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  9. 2024Dynamic metastability in the self-attention modelcited by 46
  10. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 39
  11. 2024Clustering in Causal Attention Maskingcited by 32
  12. 2025A Unified Perspective on the Dynamics of Deep Transformerscited by 32
16 more
  1. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 31
  2. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  3. 2024Measure-to-measure interpolation using Transformerscited by 29
  4. 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 26
  5. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 25
  6. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  7. 2025Critical attention scaling in long-context transformerscited by 20
  8. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17
  9. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 17
  10. 2024Clustering in pure-attention hardmax transformers and its role in sentiment analysiscited by 15
  11. 2025Synchronization of mean-field models on the circlecited by 12
  12. 2025On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformerscited by 12
  13. 2024Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformerscited by 9
  14. 2025Multistability of Self-Attention Dynamics in Transformerscited by 8
  15. 2025Attention's forward pass and Frank-Wolfecited by 6
  16. 2024On the number of modes of Gaussian kernel density estimatorscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regimecited by 5
  2. 2026Random Quadratic Form on a Sphere: Synchronization by Common Noisecited by 4
  3. 2026Propagation of Chaos in Contextual Flow Mapscited by 3
  4. 2026On the Diverse Dynamical Behaviors Arising in Deep Linear Transformerscited by 3
  5. 2026Interior interpretability with attention rollout: contraction and propagation profiles in Transformerscited by 3
  6. 2026Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformerscited by 1
  7. 2026Kinetic theory for Transformers and the lost-in-the-middle phenomenoncited by 0
  8. 2026The Role of Feed-Forward Layers in Transformer Dynamicscited by 0