# Perceptrons and localization of attention's mean-field landscape

arXiv: https://arxiv.org/abs/2601.21366 · HTML: https://mlsanity.com/paper/2601.21366/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [The emergence of clusters in self-attention dynamics](https://mlsanity.com/paper/2305.05465/) (2023)
- [Sinkformers: Transformers with Doubly Stochastic Attention](https://mlsanity.com/paper/2110.11773/) (2021)
- [Why do LLMs attend to the first token?](https://mlsanity.com/paper/2504.02732/) (2025)
- [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019)
- [Dynamic metastability in the self-attention model](https://mlsanity.com/paper/2410.06833/) (2024)
- [Emergence of meta-stable clustering in mean-field transformer models](https://mlsanity.com/paper/2410.23228/) (2024)
- [Clustering in Causal Attention Masking](https://mlsanity.com/paper/2411.04990/) (2024)
- [A Unified Perspective on the Dynamics of Deep Transformers](https://mlsanity.com/paper/2501.18322/) (2025)
- [Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems](https://mlsanity.com/paper/2109.15142/) (2021)
- [A phase transition between positional and semantic learning in a solvable model of dot-product attention](https://mlsanity.com/paper/2402.03902/) (2024)
- [Measure-to-measure interpolation using Transformers](https://mlsanity.com/paper/2411.04551/) (2024)
- [Quantitative Clustering in Mean-Field Transformer Models](https://mlsanity.com/paper/2504.14697/) (2025)
- [A multiscale analysis of mean-field transformers in the moderate interaction regime](https://mlsanity.com/paper/2509.25040/) (2025)
- [Geometric Dynamics of Signal Propagation Predict Trainability of Transformers](https://mlsanity.com/paper/2403.02579/) (2024)
- [Critical attention scaling in long-context transformers](https://mlsanity.com/paper/2510.05554/) (2025)
- [Synchronization on circles and spheres with nonlinear interactions](https://mlsanity.com/paper/2405.18273/) (2024)
- [Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation](https://mlsanity.com/paper/2505.24333/) (2025)
- [Clustering in pure-attention hardmax transformers and its role in sentiment analysis](https://mlsanity.com/paper/2407.01602/) (2024)
- [Synchronization of mean-field models on the circle](https://mlsanity.com/paper/2507.22857/) (2025)
- [On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformers](https://mlsanity.com/paper/2510.20094/) (2025)
- [Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers](https://mlsanity.com/paper/2405.15926/) (2024)
- [Multistability of Self-Attention Dynamics in Transformers](https://mlsanity.com/paper/2511.11553/) (2025)
- [Attention's forward pass and Frank-Wolfe](https://mlsanity.com/paper/2508.09628/) (2025)
- [On the number of modes of Gaussian kernel density estimators](https://mlsanity.com/paper/2412.09080/) (2024)

## Built on by (most cited first, 8 of 8)

- [Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime](https://mlsanity.com/paper/2605.10931/) (2026)
- [Random Quadratic Form on a Sphere: Synchronization by Common Noise](https://mlsanity.com/paper/2603.06187/) (2026)
- [Propagation of Chaos in Contextual Flow Maps](https://mlsanity.com/paper/2605.16747/) (2026)
- [On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers](https://mlsanity.com/paper/2607.18584/) (2026)
- [Interior interpretability with attention rollout: contraction and propagation profiles in Transformers](https://mlsanity.com/paper/2607.22367/) (2026)
- [Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers](https://mlsanity.com/paper/2605.07772/) (2026)
- [Kinetic theory for Transformers and the lost-in-the-middle phenomenon](https://mlsanity.com/paper/2605.09213/) (2026)
- [The Role of Feed-Forward Layers in Transformer Dynamics](https://mlsanity.com/paper/2609.36230/) (2026)

## Abstract (arXiv)

The forward pass of a Transformer can be seen as an interacting particle system on the unit sphere: time plays the role of layers, particles that of token embeddings, and the unit sphere idealizes layer normalization. In some weight settings the system can even be seen as a gradient flow for an explicit energy, and one can make sense of the infinite context length (mean-field) limit thanks to Wasserstein gradient flows. In this paper we study the effect of the perceptron block in this setting, and show that critical points are generically atomic and localized on subsets of the sphere.