mlsanityData to 7 Oct 2026

A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation

arXiv 2604.14722 ↗First posted 16 Apr 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Transformers commonly exhibit an attention sink: disproportionately high attention to the first position. We study this behavior in GPT-2-style models with learned query biases and absolute positional embeddings. Combining structural analysis with causal interventions, validated across natural-language, mathematical, and code inputs, we find that the sink arises from the interaction among (i) a learned query bias, (ii) the first-layer MLP transformation of the positional encoding, and (iii) structure in the key projection. Crucially, each component we identify is individually dispensable: architectures omitting each of them robustly exhibit sinks. This indicates that attention sinks may arise through distinct circuits across architectures. These findings inform mitigation of sinks, and motivate broader investigation into why sinks emerge.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 16 Apr 2026.

Builds on

19 of 28 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  8. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  9. 2025Why do LLMs attend to the first token?cited by 122
  10. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 70
  11. 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 54
  12. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
7 more
  1. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47
  2. 2025What are you sinking? A geometric approach on attention sinkcited by 26
  3. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 23
  4. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 21
  5. 2025Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sinkcited by 15
  6. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 10
  7. 2025Mamba Knockout for Unraveling Factual Information Flowcited by 4

Built on by

Not cited yet.