mlsanityData to 14 Oct 2026

Multistability of Self-Attention Dynamics in Transformers

arXiv 2511.11553 ↗First posted 14 Nov 2025cs.LG cs.SY eess.SYcited by 8

Not on the map. Why?

Abstract (arXiv)

In machine learning, a self-attention dynamics is a continuous-time multiagent-like model of the attention mechanisms of transformers. In this paper we show that such dynamics is related to a multiagent version of the Oja flow, a dynamical system that computes the principal eigenvector of a matrix corresponding for transformers to the value matrix. We classify the equilibria of the ``single-head'' self-attention system into four classes: consensus, bipartite consensus, clustering and polygonal equilibria. Multiple asymptotically stable equilibria from the first three classes often coexist in the self-attention dynamics. Interestingly, equilibria from the first two classes are always aligned with the eigenvectors of the value matrix, often but not exclusively with the principal eigenvector.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 16 of its 35 references that are in the map. First posted on arXiv 14 Nov 2025.

Builds on

16 of 35 references.

References · 16

  1. 2021Transformers in Vision: A Surveycited by 1,013
  2. 2022Transformers in Time Series: A Surveycited by 497
  3. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  4. 2021A Survey of Transformerscited by 401
  5. 2023A mathematical perspective on Transformerscited by 164
  6. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  7. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  8. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
  9. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  10. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
  11. 2024Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNscited by 34
  12. 2023Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionalscited by 32
4 more
  1. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 31
  2. 2024Setting the Record Straight on Transformer Oversmoothingcited by 21
  3. 2024The Asymptotic Behavior of Attention in Transformerscited by 16
  4. 2018A Lifting method for analyzing distributed synchronization on the unit spherecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Perceptrons and localization of attention's mean-field landscapecited by 8
  2. 2026Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regimecited by 5
  3. 2026Reachability and asymptotics of Gaussian Transformer dynamicscited by 1
  4. 2026Clustered Attractor Manifolds and Dynamical Condensation in Self-Attentioncited by 1
  5. 2026Analogies between Transformer Layers and Power Methodcited by 0
  6. 2026Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Spherecited by 0
  7. 2026Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformerscited by 0
  8. 2026Deep learning from a control vantage pointcited by 0