# The Asymptotic Behavior of Attention in Transformers arXiv: https://arxiv.org/abs/2412.02682 ยท HTML: https://mlsanity.com/paper/2412.02682/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 50 references that are in the map. - First posted on arXiv 3 Dec 2024. ## Builds on (most cited first, 28 of 28) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016) - [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021) - [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023) - [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022) - [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015) - [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021) - [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019) - [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019) - [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015) - [DeepNet: Scaling Transformers to 1,000 Layers](https://mlsanity.com/paper/2203.00555/) (2022) - [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023) - [The emergence of clusters in self-attention dynamics](https://mlsanity.com/paper/2305.05465/) (2023) - [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019) - [The rising costs of training frontier AI models](https://mlsanity.com/paper/2405.21015/) (2024) - [On Limitations of the Transformer Architecture](https://mlsanity.com/paper/2402.08164/) (2024) - [Natural Language Processing Advancements By Deep Learning: A Survey](https://mlsanity.com/paper/2003.01200/) (2020) - [On the Role of Attention Masks and LayerNorm in Transformers](https://mlsanity.com/paper/2405.18781/) (2024) - [Rank Diminishing in Deep Neural Networks](https://mlsanity.com/paper/2206.06072/) (2022) - [Dynamic metastability in the self-attention model](https://mlsanity.com/paper/2410.06833/) (2024) - [The Impact of Depth on Compositional Generalization in Transformer Language Models](https://mlsanity.com/paper/2310.19956/) (2023) - [Clustering in Causal Attention Masking](https://mlsanity.com/paper/2411.04990/) (2024) - [Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems](https://mlsanity.com/paper/2109.15142/) (2021) - [Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding](https://mlsanity.com/paper/2310.19671/) (2023) - [A Lifting method for analyzing distributed synchronization on the unit sphere](https://mlsanity.com/paper/1805.02528/) (2018) - [Consensus on Lie groups for the Riemannian Center of Mass](https://mlsanity.com/paper/2308.08054/) (2023) ## Built on by (most cited first, 16 of 16) - [Quantitative Clustering in Mean-Field Transformer Models](https://mlsanity.com/paper/2504.14697/) (2025) - [Synchronization of mean-field models on the circle](https://mlsanity.com/paper/2507.22857/) (2025) - [Multistability of Self-Attention Dynamics in Transformers](https://mlsanity.com/paper/2511.11553/) (2025) - [Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime](https://mlsanity.com/paper/2605.10931/) (2026) - [On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers](https://mlsanity.com/paper/2607.18584/) (2026) - [Localmax dynamics for attention in transformers and its asymptotic behavior](https://mlsanity.com/paper/2509.15958/) (2025) - [Context-Selective State Space Models: Feedback is All You Need](https://mlsanity.com/paper/2510.14027/) (2025) - [Dual Filter: A Transformer-like Inference Architecture for Hidden Markov Models](https://mlsanity.com/paper/2505.00818/) (2025) - [How Smoothing is N-simplicial Attention?](https://mlsanity.com/paper/2512.15600/) (2025) - [Transformer-like Inference from Optimal Control](https://mlsanity.com/paper/2605.15608/) (2026) - [Analogies between Transformer Layers and Power Method](https://mlsanity.com/paper/2605.25619/) (2026) - [Stable Transformer-Actor-Critic Model Predictive Control: A Contraction Analysis Approach](https://mlsanity.com/paper/2606.20197/) (2026) - [Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere](https://mlsanity.com/paper/2607.24502/) (2026) - [The Attention Within: Consensus Dynamics in Selective State Space Models](https://mlsanity.com/paper/2609.17997/) (2026) - [Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers](https://mlsanity.com/paper/2610.13518/) (2026) - [Deep learning from a control vantage point](https://mlsanity.com/paper/2610.14385/) (2026) ## Abstract (arXiv) The transformer architecture has become the foundation of modern Large Language Models (LLMs), yet its theoretical properties are still not well understood. As with classic neural networks, a common approach to improve these models is to increase their size and depth. However, such strategies may be suboptimal, as several works have shown that adding more layers yields increasingly diminishing returns. More importantly, prior studies have shown that increasing depth may lead to model collapse, i.e., all the tokens converge to a single cluster, undermining the ability of LLMs to generate diverse outputs. Building on differential equation models for the transformer dynamics, we prove that all the tokens in a transformer asymptotically converge to a cluster as depth increases. At the technical level we leverage tools from control theory, including consensus dynamics on manifolds and input-to-state stability (ISS). We then extend our analysis to autoregressive models, exploiting their structure to further generalize the theoretical guarantees.