mlsanityData to 14 Oct 2026

Dynamic metastability in the self-attention model

arXiv 2410.06833 ↗First posted 9 Oct 2024cs.LG math.AP math.DScited by 46

Not on the map. Why?

Abstract (arXiv)

We consider the self-attention model - an interacting particle system on the unit sphere, which serves as a toy model for Transformers, the deep neural network architecture behind the recent successes of large language models. We prove the appearance of dynamic metastability conjectured in [GLPR23] - although particles collapse to a single cluster in infinite time, they remain trapped near a configuration of several clusters for an exponentially long period of time. By leveraging a gradient flow interpretation of the system, we also connect our result to an overarching framework of slow motion of gradient flows proposed by Otto and Reznikoff [OR07] in the context of coarsening and the Allen-Cahn equation. We finally probe the dynamics beyond the exponentially long period of metastability, and illustrate that, under an appropriate time-rescaling, the energy reaches its global maximum in finite time and has a staircase profile, with trajectories manifesting saddle-to-saddle-like behavior, reminiscent of recent works in the analysis of training dynamics via gradient descent for two-layer neural networks.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 10 of its 65 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

10 of 65 references.

References · 10

  1. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  2. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 48
  3. 2024Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNscited by 34
  4. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  5. 2024Setting the Record Straight on Transformer Oversmoothingcited by 21
  6. 2024Self-attention Networks Localize When QK-eigenspectrum Concentratescited by 19
  7. 2024Clustering in pure-attention hardmax transformers and its role in sentiment analysiscited by 15
  8. 2022Expander graphs are globally synchronizingcited by 8
  9. 2024Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamicscited by 7
  10. 2024Iterated Schr"odinger bridge approximation to Wasserstein Gradient Flowscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 16; 2026: 25.

46 papers.

Sort:

By area:

  1. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  2. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 39
  3. 2024Clustering in Causal Attention Maskingcited by 32
  4. 2025The Mean-Field Dynamics of Transformerscited by 30
  5. 2024Measure-to-measure interpolation using Transformerscited by 29
  6. 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 26
  7. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 25
  8. 2025Critical attention scaling in long-context transformerscited by 20
  9. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  10. 2024The Asymptotic Behavior of Attention in Transformerscited by 16
  11. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  12. 2025Synchronization of mean-field models on the circlecited by 12
34 more
  1. 2025On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformerscited by 12
  2. 2026To Use or not to Use Muon: How Simplicity Bias in Optimizers Matterscited by 10
  3. 2025Normalization in Attention Dynamicscited by 9
  4. 2026Perceptrons and localization of attention's mean-field landscapecited by 8
  5. 2026Clustering in Deep Stochastic Transformerscited by 8
  6. 2025Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobianscited by 5
  7. 2026Homogenized Transformerscited by 5
  8. 2026Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regimecited by 5
  9. 2024On the number of modes of Gaussian kernel density estimatorscited by 4
  10. 2026Random Quadratic Form on a Sphere: Synchronization by Common Noisecited by 4
  11. 2025Provably Learning from Modern Language Models via Low Logit Rankcited by 3
  12. 2026Propagation of Chaos in Contextual Flow Mapscited by 3
  13. 2026On the Diverse Dynamical Behaviors Arising in Deep Linear Transformerscited by 3
  14. 2025Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models?cited by 2
  15. 2025Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encodingcited by 2
  16. 2026YuriiFormer: A Suite of Nesterov-Accelerated Transformerscited by 2
  17. 2026Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalizationcited by 2
  18. 2026Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attentioncited by 2
  19. 2025Attention with Trained Embeddings Provably Selects Important Tokenscited by 1
  20. 2026Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformerscited by 1
  21. 2026Reachability and asymptotics of Gaussian Transformer dynamicscited by 1
  22. 2026Clustered Attractor Manifolds and Dynamical Condensation in Self-Attentioncited by 1
  23. 2026Steering Recurrent Reasoners at Inference Time with Readout Feedbackcited by 1
  24. 2026SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understandingcited by 1
  25. 2025Revisiting Transformers with Insights from Image Filtering and Boostingcited by 0
  26. 2026Kinetic theory for Transformers and the lost-in-the-middle phenomenoncited by 0
  27. 2026A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attentioncited by 0
  28. 2026Analogies between Transformer Layers and Power Methodcited by 0
  29. 2026Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variablescited by 0
  30. 2026Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Spherecited by 0
  31. 2026Random Quadratic Form with random forcing: Metastable synchronization by noisecited by 0
  32. 2026The Role of Feed-Forward Layers in Transformer Dynamicscited by 0
  33. 2026Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformerscited by 0
  34. 2026Deep learning from a control vantage pointcited by 0