mlsanityData to 14 Oct 2026

Why do LLMs attend to the first token?

arXiv 2504.02732 ↗First posted 3 Apr 2025cs.CLcited by 127

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) tend to attend heavily to the first token in the sequence -- creating a so-called attention sink. Many works have studied this phenomenon in detail, proposing various ways to either leverage or alleviate it. Attention sinks have been connected to quantisation difficulties, security issues, and streaming attention. Yet, while many works have provided conditions in which they occur or not, a critical question remains shallowly answered: Why do LLMs learn such patterns and how are they being used? In this work, we argue theoretically and empirically that this mechanism provides a method for LLMs to avoid over-mixing, connecting this to existing lines of work that study mathematically how information propagates in Transformers. We conduct experiments to validate our theoretical intuitions and show how choices such as context length, depth, and data packing influence the sink behaviour. We hope that this study provides a new practical perspective on why attention sinks are useful in LLMs, leading to a better understanding of the attention patterns that form during training.

Key facts. Cited by 127 papers in the map (data through 14 Oct 2026). Builds on 27 of its 34 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

27 of 34 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  7. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  8. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  9. 2024Massive Activations in Large Language Modelscited by 294
  10. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  11. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  12. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
15 more
  1. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  2. 2023A mathematical perspective on Transformerscited by 164
  3. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  4. 2019On Identifiability in Transformerscited by 116
  5. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
  6. 2022Not too little, not too much: a theoretical analysis of graph (over)smoothingcited by 106
  7. 2024Transformers need glasses! Information over-squashing in language taskscited by 91
  8. 2022Understanding convolution on graphs via energiescited by 59
  9. 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 56
  10. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
  11. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  12. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 48
  13. 2025On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learningcited by 43
  14. 2024Analysing The Impact of Sequence Composition on Language Model Pre-Trainingcited by 25
  15. 2025What makes a good feedforward computational graph?cited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 49; 2026: 78.

127 papers.

Sort:

By area:

  1. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 52
  2. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 51
  3. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 33
  4. 2025Qronos: Correcting the Past by Shaping the Future... in Post-Training Quantizationcited by 28
  5. 2025What are you sinking? A geometric approach on attention sinkcited by 28
  6. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 25
  7. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmaxcited by 22
  8. 2025To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Modelscited by 20
  9. 2025Attention Sinks in Diffusion Language Modelscited by 20
  10. 2025Robust Hallucination Detection in LLMs via Adaptive Token Selectioncited by 19
  11. 2025Hallucination Detection in LLMs with Topological Divergence on Attention Graphscited by 19
  12. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 18
38 more
  1. 2025Lost at the Beginning of Reasoningcited by 15
  2. 2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Modelscited by 13
  3. 2025ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Trainingcited by 12
  4. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  5. 2025Return of ChebNet: Understanding and Improving an Overlooked GNN on Long Range Taskscited by 12
  6. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 11
  7. 2025Attention Consistency for LLMs Explanationcited by 11
  8. 2025Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMscited by 11
  9. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 11
  10. 2025MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoningcited by 10
  11. 2025Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformerscited by 8
  12. 2025Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformerscited by 8
  13. 2026Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruningcited by 8
  14. 2026Perceptrons and localization of attention's mean-field landscapecited by 8
  15. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  16. 2025Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretabilitycited by 7
  17. 2025The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilitiescited by 7
  18. 2025From Reasoning to Answer: Empirical, Attention-Based and Mechanistic Insights into Distilled DeepSeek R1 Modelscited by 6
  19. 2025Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearningcited by 6
  20. 2026FACTUM: Mechanistic Detection of Citation Hallucination in Long-Form RAGcited by 6
  21. 2025Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMscited by 5
  22. 2026Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapsecited by 5
  23. 2025CTR-Sink: Attention Sink for Language Models in Click-Through Rate Predictioncited by 4
  24. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 4
  25. 2026One Token Is Enough: Improving Diffusion Language Models with a Sink Tokencited by 4
  26. 2026Semantic Routing: Exploring Multi-Layer LLM Feature Weighting for Diffusion Transformerscited by 4
  27. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  28. 2026Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformerscited by 4
  29. 2025LayerCake: Token-Aware Contrastive Decoding within Large Language Model Layerscited by 3
  30. 2025A Large-Scale Benchmark for Evaluating Large Language Models on Medical Question Answering in Romaniancited by 3
  31. 2026What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Modelscited by 3
  32. 2026What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMscited by 3
  33. 2026Effective Distillation to Hybrid xLSTM Architecturescited by 3
  34. 2026Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformerscited by 3
  35. 2026Frayed RoPE and Long Inputs: A Geometric Perspectivecited by 3
  36. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Informationcited by 3
  37. 2026The Truth Lies Somewhere in the Middle (of the Generated Tokens)cited by 3
  38. 2026OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruningcited by 3

50 most cited of 127 All 127 (JSON)