mlsanityData to 14 Oct 2026

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks

arXiv 2603.05498 ↗First posted 5 Mar 2026cs.AI cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

We study two recurring phenomena in Transformer language models: massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks, in which certain tokens attract disproportionate attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same tokens, but their functional roles and causal relationship remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally: they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters of the model. Attention sinks operate locally: they modulate attention outputs across heads and bias individual heads toward short-range dependencies. We identify the pre-norm configuration as the key choice that enables the co-occurrence, and show that ablating it causes the two phenomena to decouple.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 67 of its 81 references that are in the map. First posted on arXiv 5 Mar 2026.

Builds on

67 of 81 references.

References · 67

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2025Qwen3 Technical Reportcited by 8,820
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  12. 2020GLU Variants Improve Transformercited by 1,804
38 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,480
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  3. 2020On Layer Normalization in the Transformer Architecturecited by 922
  4. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  6. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  7. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  9. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  10. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  11. 2024Massive Activations in Large Language Modelscited by 294
  12. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  13. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  14. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  15. 2019Understanding and Improving Layer Normalizationcited by 217
  16. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  17. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  18. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  19. 2026Ministral 3cited by 114
  20. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  21. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
  22. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
  23. 2025Pretraining Large Language Models with NVFP4cited by 76
  24. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 74
  25. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  26. 2024Theory, Analysis, and Best Practices for Sigmoid Self-Attentioncited by 64
  27. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 63
  28. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  29. 2023Training Transformers with 4-bit Integerscited by 60
  30. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 59
  31. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
  32. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  33. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  34. 2024The Super Weight in Large Language Modelscited by 52
  35. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 52
  36. 2023Intriguing Properties of Quantization at Scalecited by 49
  37. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 46
  38. 2024Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantizationcited by 33

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 27.

27 papers.

Sort:

By area:

  1. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 11
  2. 2026Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigationcited by 11
  3. 2026Rethinking the Role of Tensor Decompositions in Post-Training LLM Compressioncited by 4
  4. 2026Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformerscited by 3
  5. 2026Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantizationcited by 2
  6. 2026Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateauscited by 2
  7. 2026Bug or Feature2: Weight Drift, Activation Sparsity and Spikescited by 1
  8. 2026Massive Activations Are Architecturally Robust: A Controlled Scratch/Commitment Residual Stream Testcited by 1
  9. 2026Hybrid Gated Attentioncited by 1
  10. 2026Trust the Mass: Forced Weights in KV-Cache Evictioncited by 1
  11. 2026A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigationcited by 0
  12. 2026Attractor Geometry of Transformer Memory: From Conflict Arbitration to Confident Hallucinationcited by 0
15 more
  1. 2026Measuring Maximum Activations in Open Large Language Modelscited by 0
  2. 2026When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Modelscited by 0
  3. 2026Information-Regularized Attention for Visual-Centric Reasoningcited by 0
  4. 2026Harrison.Rad 1.5 Technical Report: A radiology foundation model that can draft reports from images, priors and clinical contextcited by 0
  5. 2026SharedSAE: One Feature Dictionary Across Language Modelscited by 0
  6. 2026Contrastive Projection: Reading Transformer Internals by Differencing Logit Lensescited by 0
  7. 2026Affinity-Aware Sharding for Delayed Tensor Parallelismcited by 0
  8. 2026Structure is not mechanism: high-gain gated-FFN rows across text and genomic foundation modelscited by 0
  9. 2026Abstention and Noise Filtering: Two Missing Primitives of Softmax Attentioncited by 0
  10. 2026Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Directioncited by 0
  11. 2026Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Modelscited by 0
  12. 2026Which the Eye Fears: Writing with Read-Blindness Explains Massive Activations in Transformerscited by 0
  13. 2026Selecting The Most Informative Tokens in Natural Language Autoencoderscited by 0
  14. 2026Spectral Geometry of Attention: From Information Routing to Uncertaintycited by 0
  15. 2026Massive Activation Gating Channel in Large Language Modelscited by 0