mlsanityData to 14 Oct 2026

Clustering in Causal Attention Masking

arXiv 2411.04990 ↗First posted 7 Nov 2024cs.LG cs.AI math.APcited by 32

Not on the map. Why?

Abstract (arXiv)

This work presents a modification of the self-attention dynamics proposed by Geshkovski et al. (arXiv:2312.10794) to better reflect the practically relevant, causally masked attention used in transformer architectures for generative AI. This modification translates into an interacting particle system that cannot be interpreted as a mean-field gradient flow. Despite this loss of structure, we significantly strengthen the results of Geshkovski et al. (arXiv:2312.10794) in this context: While previous rigorous results focused on cases where all three matrices (Key, Query, and Value) were scaled identities, we prove asymptotic convergence to a single cluster for arbitrary key-query matrices and a value matrix equal to the identity. Additionally, we establish a connection to the classical R'enyi parking problem from combinatorial geometry to make initial theoretical steps towards demonstrating the existence of meta-stable states.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

21 of 38 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  5. 2019Root Mean Square Layer Normalizationcited by 1,480
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  7. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  8. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  9. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  10. 2023A mathematical perspective on Transformerscited by 164
  11. 2023The emergence of clusters in self-attention dynamicscited by 140
  12. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
9 more
  1. 2023Demystifying Oversmoothing in Attention-Based Graph Neural Networkscited by 49
  2. 2024Dynamic metastability in the self-attention modelcited by 46
  3. 2023How Smooth Is Attention?cited by 44
  4. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 39
  5. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  6. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17
  7. 2021On the Regularity of Attentioncited by 11
  8. 2024Generic controllability of equivariant systems and applications to particle systems and neural networkscited by 9
  9. 2022Expander graphs are globally synchronizingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 17.

32 papers.

Sort:

  1. 2025The Mean-Field Dynamics of Transformerscited by 30
  2. 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 26
  3. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 25
  4. 2025Critical attention scaling in long-context transformerscited by 20
  5. 2024The Asymptotic Behavior of Attention in Transformerscited by 16
  6. 2025Synchronization of mean-field models on the circlecited by 12
  7. 2025On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformerscited by 12
  8. 2025Normalization in Attention Dynamicscited by 9
  9. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  10. 2026Perceptrons and localization of attention's mean-field landscapecited by 8
  11. 2026Clustering in Deep Stochastic Transformerscited by 8
  12. 2025OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularizationcited by 7
20 more
  1. 2025Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobianscited by 5
  2. 2026Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regimecited by 5
  3. 2024On the number of modes of Gaussian kernel density estimatorscited by 4
  4. 2025Token Sample Complexity of Attentioncited by 3
  5. 2025Exact Sequence Interpolation with Transformerscited by 2
  6. 2026Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attentioncited by 2
  7. 2026Diffusion Operator Geometry of Feedforward Representationscited by 1
  8. 2026Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformerscited by 1
  9. 2026Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing preventioncited by 1
  10. 2026Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flowscited by 1
  11. 2026Attention as Frustrated Synchronizationcited by 1
  12. 2026Feed-Forward Steering in Transformer Residual Dynamicscited by 1
  13. 2026Faster Query-Key Learning Sharpens Attention in Self-Attention Modelscited by 1
  14. 2026Clustered Attractor Manifolds and Dynamical Condensation in Self-Attentioncited by 1
  15. 2025Multispin Physics of AI Tipping Points and Hallucinationscited by 0
  16. 2026Kinetic theory for Transformers and the lost-in-the-middle phenomenoncited by 0
  17. 2026A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attentioncited by 0
  18. 2026Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Spherecited by 0
  19. 2026Opinion Leader Dynamics: How Sparse Attention Shapes Token Clusteringcited by 0
  20. 2026Understanding Clustering in Slot Attention via Particle Dynamicscited by 0