mlsanityData to 14 Oct 2026

When Attention Sink Emerges in Language Models: An Empirical View

arXiv 2410.10781 ↗First posted 14 Oct 2024cs.CL cs.AI cs.LGcited by 208

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Language Models (LMs) assign significant attention to the first token, even if it is not semantically important, which is known as attention sink. This phenomenon has been widely adopted in applications such as streaming/long context generation, KV cache optimization, inference acceleration, model quantization, and others. Despite its widespread use, a deep understanding of attention sink in LMs is still lacking. In this work, we first demonstrate that attention sinks exist universally in LMs with various inputs, even in small models. Furthermore, attention sink is observed to emerge during the LM pre-training, motivating us to investigate how optimization, data distribution, loss function, and model architecture in LM pre-training influence its emergence. We highlight that attention sink emerges after effective optimization on sufficient training data. The sink position is highly correlated with the loss function and data distribution. Most importantly, we find that attention sink acts more like key biases, storing extra attention scores, which could be non-informative and not contribute to the value computation. We also observe that this phenomenon (at least partially) stems from tokens' inner dependence on attention scores as a result of softmax normalization. After relaxing such dependence by replacing softmax attention with other attention operations, such as sigmoid attention without normalization, attention sinks do not emerge in LMs up to 1B parameters. The code is available at https://github.com/sail-sg/Attention-Sink.

Key facts. Cited by 208 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 54 of its 66 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

54 of 66 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference26 Jun
  2. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
  3. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  4. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models17 May
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct

Other references · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2024The Llama 3 Herd of Modelscited by 14,715
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2016Layer Normalizationcited by 5,669
  11. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
37 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2023Mistral 7Bcited by 3,164
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  10. 2020GLU Variants Improve Transformercited by 1,804
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  12. 2019Root Mean Square Layer Normalizationcited by 1,480
  13. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  14. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  15. 2023Vision Transformers Need Registerscited by 871
  16. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  17. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  18. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  19. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  20. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  21. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  22. 2024Massive Activations in Large Language Modelscited by 294
  23. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  24. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  25. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  26. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  27. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  28. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 150
  29. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  30. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 74
  31. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 73
  32. 2024Theory, Analysis, and Best Practices for Sigmoid Self-Attentioncited by 64
  33. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 63
  34. 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 56
  35. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  36. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scalecited by 50
  37. 2024StableMask: Refining Causal Masking in Decoder-only Transformercited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 72; 2026: 130.

208 papers.

Sort:

By area:

  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  2. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  3. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  4. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  5. 2026MiMo-V2-Flash Technical Reportcited by 191
  6. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  7. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  8. 2025Why do LLMs attend to the first token?cited by 127
  9. 2025On the Emergence of Position Bias in Transformerscited by 59
  10. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
  11. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 52
  12. 2025Vision Transformers Don't Need Trained Registerscited by 41
38 more
  1. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 39
  2. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 33
  3. 2025What are you sinking? A geometric approach on attention sinkcited by 28
  4. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 27
  5. 2024Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Modelscited by 26
  6. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 25
  7. 2025Sliding Window Attention Training for Efficient Large Language Modelscited by 24
  8. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 23
  9. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmaxcited by 22
  10. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 21
  11. 2025BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Modelscited by 20
  12. 2025To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Modelscited by 20
  13. 2025Attention Sinks in Diffusion Language Modelscited by 20
  14. 2026Efficient Autoregressive Video Diffusion with Dummy Headcited by 20
  15. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 18
  16. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 18
  17. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 17
  18. 2025Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddingscited by 16
  19. 2025Interpreting the Repeated Token Phenomenon in Large Language Modelscited by 16
  20. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  21. 2025Analysis of Attention in Video Diffusion Transformerscited by 15
  22. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 14
  23. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 12
  24. 2024Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Mapscited by 12
  25. 2025Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inferencecited by 12
  26. 2025ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Trainingcited by 12
  27. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 11
  28. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 11
  29. 2025Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMscited by 11
  30. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 11
  31. 2025Artifacts and Attention Sinks: Structured Approximations for Efficient Vision Transformerscited by 10
  32. 2025HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputscited by 9
  33. 2025Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMscited by 9
  34. 2026HorizonStream: Long-Horizon Attention for Streaming 3D Reconstructioncited by 9
  35. 2025Identifying and Evaluating Inactive Heads in Pretrained LLMscited by 8
  36. 2025OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inferencecited by 8
  37. 2025Scaling Sequence-to-Sequence Generative Neural Renderingcited by 8
  38. 2025Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformerscited by 8

50 most cited of 208 All 208 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
  2. 2026Summarize Before You Speak with ARACH: A Training-Free Inference-Time Plug-In for Enhancing LLMs via Global Attention Reallocation
  3. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  4. 2026Sliding-window beats linear attention
  5. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  6. 2024From Attention to Activation: Unravelling the Enigmas of Large Language Models
  7. 2025Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models
  8. 2025Training-free Context-adaptive Attention for Efficient Long Context Modeling