# What are you sinking? A geometric approach on attention sink

arXiv: https://arxiv.org/abs/2508.02546 · HTML: https://mlsanity.com/paper/2508.02546/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Builds on (most cited first, 17 of 17)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [When Attention Sink Emerges in Language Models: An Empirical View](https://mlsanity.com/paper/2410.10781/) (2024)
- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)
- [Topological Autoencoders](https://mlsanity.com/paper/1906.00722/) (2019)
- [Why do LLMs attend to the first token?](https://mlsanity.com/paper/2504.02732/) (2025)
- [Topology of deep neural networks](https://mlsanity.com/paper/2004.06093/) (2020)
- [Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration](https://mlsanity.com/paper/2406.15765/) (2024)
- [Spectral Filters, Dark Signals, and Attention Sinks](https://mlsanity.com/paper/2402.09221/) (2024)
- [Quantifying Context Mixing in Transformers](https://mlsanity.com/paper/2301.12971/) (2023)
- [Learning High-Frequency Functions Made Easy with Sinusoidal Positional Encoding](https://mlsanity.com/paper/2407.09370/) (2024)
- [Beyond Position: the emergence of wavelet-like properties in Transformers](https://mlsanity.com/paper/2410.18067/) (2024)

## Built on by (most cited first, 28 of 28)

- [Attention Sinks in Diffusion Language Models](https://mlsanity.com/paper/2510.15731/) (2025)
- [SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers](https://mlsanity.com/paper/2609.01343/) (2026)
- [Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks](https://mlsanity.com/paper/2603.11487/) (2026)
- [Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing](https://mlsanity.com/paper/2602.02159/) (2026)
- [On the Existence and Behavior of Secondary Attention Sinks](https://mlsanity.com/paper/2512.22213/) (2025)
- [Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sink](https://mlsanity.com/paper/2602.05228/) (2026)
- [When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models](https://mlsanity.com/paper/2604.03316/) (2026)
- [Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention](https://mlsanity.com/paper/2602.23057/) (2026)
- [NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks](https://mlsanity.com/paper/2603.06922/) (2026)
- [SinkTrack: Attention Sink based Context Anchoring for Large Language Models](https://mlsanity.com/paper/2604.10027/) (2026)
- [A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models](https://mlsanity.com/paper/2605.08504/) (2026)
- [Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws](https://mlsanity.com/paper/2605.21803/) (2026)
- [Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization](https://mlsanity.com/paper/2606.02288/) (2026)
- [Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation](https://mlsanity.com/paper/2604.22783/) (2026)
- [Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention](https://mlsanity.com/paper/2605.08453/) (2026)
- [Retrieval and competition: how a protein foundation model starts a protein](https://mlsanity.com/paper/2605.16331/) (2026)
- [Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference](https://mlsanity.com/paper/2607.13205/) (2026)
- [The Geometric Cost of Normalization: Affine Bounds on the Bayesian Complexity of Neural Networks](https://mlsanity.com/paper/2603.27432/) (2026)
- [Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models](https://mlsanity.com/paper/2604.10697/) (2026)
- [A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation](https://mlsanity.com/paper/2604.14722/) (2026)
- [ASAP: Attention Sink Anchored Pruning](https://mlsanity.com/paper/2605.22372/) (2026)
- [Where does Absolute Position come from in decoder-only Transformers?](https://mlsanity.com/paper/2606.06160/) (2026)
- [A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions](https://mlsanity.com/paper/2606.08105/) (2026)
- [Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention](https://mlsanity.com/paper/2609.22005/) (2026)
- [ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs](https://mlsanity.com/paper/2609.23314/) (2026)
- [Global Ranks Survive, Selected Heads Shift: BOS-Sink Topology under 4-bit Weight-Only Quantization](https://mlsanity.com/paper/2609.23585/) (2026)
- [Spectral Geometry of Attention: From Information Routing to Uncertainty](https://mlsanity.com/paper/2610.06012/) (2026)
- [Before Attention Sinks Form: Hidden-State Geometry Reveals and Modulates Their Formation](https://mlsanity.com/paper/2610.16745/) (2026)

## Abstract (arXiv)

Attention sink (AS) is a consistent pattern in transformer attention maps where certain tokens (often special tokens or positional anchors) disproportionately attract attention from other tokens. We show that in transformers, AS is not an architectural artifact, but it is the manifestation of a fundamental geometric principle: the establishment of reference frames that anchor representational spaces. We analyze several architectures and identify three distinct reference frame types, centralized, distributed, and bidirectional, that correlate with the attention sink phenomenon. We show that they emerge during the earliest stages of training as optimal solutions to the problem of establishing stable coordinate systems in high-dimensional spaces. We show the influence of architecture components, particularly position encoding implementations, on the specific type of reference frame. This perspective transforms our understanding of transformer attention mechanisms and provides insights for both architecture design and the relationship with AS.