Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
Not on the map. Why?
Abstract (arXiv)
This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represented, showing a necessary alignment between the embedding of the sink and all other embeddings. Next, we refine the current understanding of the role of sinks in oversmoothing prevention: we specify the conditions under which dense attention provably smooths more than sparse attention, and empirically verify that such conditions are often satisfied in practice. We further prove an equivalence between sinks and hard attention switch, in which the output of the attention is identically 0. Finally, we relax the hard attention switch by allowing token self-communication: we provide a quantitative comparison of the costs of representing sinks vs. diagonal patterns, showing why sinks are favored in pretrained transformers. The introduction and analysis of diagonal patterns and the generalization of the attention switch close the gap between what oversmoothing prevention requires and what sinks provide, while also establishing when and why attention layers act like MLPs if token communication is not necessary.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 38 of its 42 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
38 of 42 references.
References · 38
- 2017Attention Is All You Needcited by 60,361
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2023Vision Transformers Need Registerscited by 871
- 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
- 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
- 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
- 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
- 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
- 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
- 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
26 more
- 2023The emergence of clusters in self-attention dynamicscited by 140
- 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
- 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
- 2025Why do LLMs attend to the first token?cited by 127
- 2024Transformers need glasses! Information over-squashing in language taskscited by 91
- 2024Transformers, parallel computation, and logarithmic depthcited by 79
- 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 74
- 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
- 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
- 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 52
- 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 48
- 2024Clustering in Causal Attention Maskingcited by 32
- 2025What are you sinking? A geometric approach on attention sinkcited by 28
- 2024Weight decay induces low-rank attention layerscited by 27
- 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
- 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 23
- 2024Setting the Record Straight on Transformer Oversmoothingcited by 21
- 2025To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Modelscited by 20
- 2025Attention Sinks in Diffusion Language Modelscited by 20
- 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 17
- 2025Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sinkcited by 15
- 2025Analysis of Attention in Video Diffusion Transformerscited by 15
- 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 11
- 2024StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Lossescited by 9
- 2026Exclusive Self Attentioncited by 6
- 2026Stochastic Parroting in Temporal Attention -- Regulating the Diagonal Sinkcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.