mlsanityData to 13 Oct 2026

What are you sinking? A geometric approach on attention sink

arXiv 2508.02546 ↗First posted 4 Aug 2025cs.LG cs.AI cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

Attention sink (AS) is a consistent pattern in transformer attention maps where certain tokens (often special tokens or positional anchors) disproportionately attract attention from other tokens. We show that in transformers, AS is not an architectural artifact, but it is the manifestation of a fundamental geometric principle: the establishment of reference frames that anchor representational spaces. We analyze several architectures and identify three distinct reference frame types, centralized, distributed, and bidirectional, that correlate with the attention sink phenomenon. We show that they emerge during the earliest stages of training as optimal solutions to the problem of establishing stable coordinate systems in high-dimensional spaces. We show the influence of architecture components, particularly position encoding implementations, on the specific type of reference frame. This perspective transforms our understanding of transformer attention mechanisms and provides insights for both architecture design and the relationship with AS.

Key facts. Cited by 27 papers in the map (data through 13 Oct 2026). Builds on 17 of its 23 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

17 of 23 references.

References · 17

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,628
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,093
  5. 2023Vision Transformers Need Registerscited by 850
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 849
  7. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 322
  8. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 202
  9. 2022Relative representations enable zero-shot latent space communicationcited by 181
  10. 2019Topological Autoencoderscited by 126
  11. 2025Why do LLMs attend to the first token?cited by 123
  12. 2020Topology of deep neural networkscited by 111
5 more
  1. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 72
  2. 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 55
  3. 2023Quantifying Context Mixing in Transformerscited by 43
  4. 2024Learning High-Frequency Functions Made Easy with Sinusoidal Positional Encodingcited by 8
  5. 2024Beyond Position: the emergence of wavelet-like properties in Transformerscited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 2; 2026: 25.

27 papers.

Sort:

By area:

  1. 2025Attention Sinks in Diffusion Language Modelscited by 17
  2. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 16
  3. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 10
  4. 2026Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusingcited by 6
  5. 2025On the Existence and Behavior of Secondary Attention Sinkscited by 3
  6. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
  7. 2026Affine-Scaled Attention: Towards Flexible and Stable Transformer Attentioncited by 3
  8. 2026NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networkscited by 3
  9. 2026When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Modelscited by 2
  10. 2026A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Modelscited by 2
  11. 2026Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Lawscited by 2
  12. 2026Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantizationcited by 2
15 more
  1. 2026SinkTrack: Attention Sink based Context Anchoring for Large Language Modelscited by 1
  2. 2026Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptationcited by 1
  3. 2026Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing preventioncited by 1
  4. 2026Retrieval and competition: how a protein foundation model starts a proteincited by 1
  5. 2026Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inferencecited by 1
  6. 2026The Geometric Cost of Normalization: Affine Bounds on the Bayesian Complexity of Neural Networkscited by 0
  7. 2026Attention Sinks as Internal Signals for Hallucination Detection in Large Language Modelscited by 0
  8. 2026A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigationcited by 0
  9. 2026ASAP: Attention Sink Anchored Pruningcited by 0
  10. 2026Where does Absolute Position come from in decoder-only Transformers?cited by 0
  11. 2026A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventionscited by 0
  12. 2026Abstention and Noise Filtering: Two Missing Primitives of Softmax Attentioncited by 0
  13. 2026ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMscited by 0
  14. 2026Global Ranks Survive, Selected Heads Shift: BOS-Sink Topology under 4-bit Weight-Only Quantizationcited by 0
  15. 2026Spectral Geometry of Attention: From Information Routing to Uncertaintycited by 0