mlsanityData to 14 Oct 2026

Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers

arXiv 2610.13518 ↗First posted 9 Oct 2026math.DS cs.LG cs.SYcited by 0

Not on the map. Why?

Abstract (arXiv)

In machine learning, a self-attention dynamics describes the attention mechanism of transformers, and is obtained treating the layer index of a transformer as time index. Even though naturally discrete-time, this dynamical system is most often studied in continuous-time. Scope of this paper is to provide a thorough investigation of the discrete-time version of the self-attention dynamics: a complete classification of its equilibria is given, and their stability properties are studied, in both the single-head and the multihead case. We show that in both cases the main attractor of the dynamics is a consensus equilibrium aligned with the principal eigenvector of the value matrix (in the single-head case, a corresponding matrix of weights in the multihead case), even though other locally asymptotically stable bipartite equilibria can be show to exist at least in the single-head situation. The behavior is confirmed by looking at the learned weight matrices of a real transformer.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 24 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

21 of 24 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  3. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  4. 2023A mathematical perspective on Transformerscited by 164
  5. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  6. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  7. 2023The emergence of clusters in self-attention dynamicscited by 140
  8. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
  9. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  10. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
  11. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 48
  12. 2021On the Expressive Power of Self-Attention Matricescited by 47
9 more
  1. 2024Dynamic metastability in the self-attention modelcited by 46
  2. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 39
  3. 2024Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNscited by 34
  4. 2023Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionalscited by 32
  5. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 31
  6. 2024Setting the Record Straight on Transformer Oversmoothingcited by 21
  7. 2024The Asymptotic Behavior of Attention in Transformerscited by 16
  8. 2025Multistability of Self-Attention Dynamics in Transformerscited by 8
  9. 2025A Theoretical Study of (Hyper) Self-Attention through the Lens of Interactions: Representation, Training, Generalizationcited by 5

Built on by

Not cited yet.