mlsanityData to 14 Oct 2026

It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention

arXiv 2609.09085 ↗First posted 8 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) often exhibit "Attention Sink" (AS) and the accompanying "Massive Activations" (MAs) at the initial position of a sequence. These phenomena frequently co-occur, and MAs can pose challenges for low-bit quantization. In this study, we analyze the factors underlying AS and MAs that emerge at the initial position regardless of the token occupying it. Our experiments suggest that self-concentration of attention, resulting from the causal mask, and the subsequent Value-non-mixing in attention outputs contribute to AS and MAs. These findings provide new empirical evidence on the internal dynamics of LLMs, offering insights that may inform future quantization strategies and advance our understanding of the internal mechanisms of attention layers.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 25 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

15 of 25 references.

References · 15

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  8. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  9. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  10. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  11. 2025Why do LLMs attend to the first token?cited by 127
  12. 2025On the Emergence of Position Bias in Transformerscited by 59
3 more
  1. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 52
  2. 2024House of Cards: Massive Weights in LLMscited by 9
  3. 2025Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024When Attention Sink Emerges in Language Models: An Empirical View
  2. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs
  3. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs
  4. 2026Attention Sinks in Diffusion Transformers: A Causal Analysis
  5. 2025On the Existence and Behavior of Secondary Attention Sinks
  6. 2025Attention Sinks in Diffusion Language Models
  7. 2026A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models
  8. 2026When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction