# Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation arXiv: https://arxiv.org/abs/2604.10098 ยท HTML: https://mlsanity.com/paper/2604.10098/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 104 of its 245 references that are in the map. - First posted on arXiv 11 Apr 2026. ## Builds on (most cited first, 30 of 104) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016) - [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023) - [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023) - [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020) - [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025) - [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025) - [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024) - [LongLive: Real-time Interactive Long Video Generation](https://mlsanity.com/paper/2509.22622/) (2025) - [Kimi Linear: An Expressive, Efficient Attention Architecture](https://mlsanity.com/paper/2510.26692/) (2025) - [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026) - [Streaming 4D Visual Geometry Transformer](https://mlsanity.com/paper/2507.11539/) (2025) - [Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation](https://mlsanity.com/paper/2512.04678/) (2025) - [Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression](https://mlsanity.com/paper/2512.05081/) (2025) - [LongCat-Flash Technical Report](https://mlsanity.com/paper/2509.01322/) (2025) - [Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs](https://mlsanity.com/paper/2410.13835/) (2024) - [The Super Weight in Large Language Models](https://mlsanity.com/paper/2411.07191/) (2024) - [Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs](https://mlsanity.com/paper/2411.09968/) (2024) - [Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding](https://mlsanity.com/paper/2502.01563/) (2025) - [LongCat-Flash-Thinking-2601 Technical Report](https://mlsanity.com/paper/2601.16725/) (2026) - [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026) - [The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks](https://mlsanity.com/paper/2603.05498/) (2026) - [LongCat-Flash-Omni Technical Report](https://mlsanity.com/paper/2511.00279/) (2025) - [ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Training](https://mlsanity.com/paper/2603.04385/) (2026) - [Sliding Window Attention Training for Efficient Large Language Models](https://mlsanity.com/paper/2502.18845/) (2025) - [A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training](https://mlsanity.com/paper/2601.22966/) (2026) - [LongStream: Long-Sequence Streaming Autoregressive Visual Geometry](https://mlsanity.com/paper/2602.13172/) (2026) - [Softpick: No Attention Sink, No Massive Activations with Rectified Softmax](https://mlsanity.com/paper/2504.20966/) (2025) ## Built on by (most cited first, 11 of 11) - [When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression](https://mlsanity.com/paper/2605.08234/) (2026) - [Priming: Hybrid State Space Models From Pre-trained Transformers](https://mlsanity.com/paper/2605.08301/) (2026) - [OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond](https://mlsanity.com/paper/2605.19660/) (2026) - [Registers Matter for Pixel-Space Diffusion Transformers](https://mlsanity.com/paper/2605.16147/) (2026) - [Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus](https://mlsanity.com/paper/2608.12149/) (2026) - [Hybrid Gated Attention](https://mlsanity.com/paper/2608.11805/) (2026) - [Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention](https://mlsanity.com/paper/2609.22005/) (2026) - [ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs](https://mlsanity.com/paper/2609.23314/) (2026) - [Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Models](https://mlsanity.com/paper/2609.32808/) (2026) - [TSGate: Timestep-Aware Gated Attention for Diffusion Transformers](https://mlsanity.com/paper/2609.34539/) (2026) - [Exploring Attention Sinks in Text-to-Image Transformers from Diffusion and Autoregressive Models](https://mlsanity.com/paper/2610.13337/) (2026) ## Abstract (arXiv) As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains. Despite their transformative impact, a persistent challenge across various Transformers is Attention Sink (AS), in which a disproportionate amount of attention is focused on a small subset of specific yet uninformative tokens. AS complicates interpretability, significantly affecting the training and inference dynamics, and exacerbates issues such as hallucinations. In recent years, substantial research has been dedicated to understanding and harnessing AS. However, a comprehensive survey that systematically consolidates AS-related research and offers guidance for future advancements remains lacking. To address this gap, we present the first survey on AS, structured around three key dimensions that define the current research landscape: Fundamental Utilization, Mechanistic Interpretation, and Strategic Mitigation. Our work makes a pivotal contribution by highlighting the key concepts and main trends in the field, guiding researchers through the evolution of AS-related studies. We envision this survey as a valuable resource, empowering researchers to effectively manage AS within the current Transformer paradigm, while simultaneously inspiring innovative advancements for the next generation of Transformers. The paper list of this work is available at https://github.com/ZunhaiSu/Awesome-Attention-Sink.