# The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks

arXiv: https://arxiv.org/abs/2603.05498 · HTML: https://mlsanity.com/paper/2603.05498/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 81 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 30 of 67)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [DataComp-LM: In search of the next generation of training sets for language models](https://mlsanity.com/paper/2406.11794/) (2024)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://mlsanity.com/paper/2410.10819/) (2024)
- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [Understanding and Improving Layer Normalization](https://mlsanity.com/paper/1911.07013/) (2019)
- [When Attention Sink Emerges in Language Models: An Empirical View](https://mlsanity.com/paper/2410.10781/) (2024)
- [Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models](https://mlsanity.com/paper/2209.13325/) (2022)
- [Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing](https://mlsanity.com/paper/2306.12929/) (2023)

## Built on by (most cited first, 27 of 27)

- [Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks](https://mlsanity.com/paper/2603.11487/) (2026)
- [Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation](https://mlsanity.com/paper/2604.10098/) (2026)
- [Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression](https://mlsanity.com/paper/2606.03465/) (2026)
- [Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers](https://mlsanity.com/paper/2603.17771/) (2026)
- [Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization](https://mlsanity.com/paper/2606.02288/) (2026)
- [Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus](https://mlsanity.com/paper/2608.12149/) (2026)
- [Bug or Feature2: Weight Drift, Activation Sparsity and Spikes](https://mlsanity.com/paper/2605.17659/) (2026)
- [Massive Activations Are Architecturally Robust: A Controlled Scratch/Commitment Residual Stream Test](https://mlsanity.com/paper/2606.20743/) (2026)
- [Hybrid Gated Attention](https://mlsanity.com/paper/2608.11805/) (2026)
- [Trust the Mass: Forced Weights in KV-Cache Eviction](https://mlsanity.com/paper/2608.25230/) (2026)
- [A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation](https://mlsanity.com/paper/2604.14722/) (2026)
- [Attractor Geometry of Transformer Memory: From Conflict Arbitration to Confident Hallucination](https://mlsanity.com/paper/2605.05686/) (2026)
- [Measuring Maximum Activations in Open Large Language Models](https://mlsanity.com/paper/2605.15572/) (2026)
- [When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models](https://mlsanity.com/paper/2606.03712/) (2026)
- [Information-Regularized Attention for Visual-Centric Reasoning](https://mlsanity.com/paper/2607.00434/) (2026)
- [Harrison.Rad 1.5 Technical Report: A radiology foundation model that can draft reports from images, priors and clinical context](https://mlsanity.com/paper/2607.05880/) (2026)
- [SharedSAE: One Feature Dictionary Across Language Models](https://mlsanity.com/paper/2609.04344/) (2026)
- [Contrastive Projection: Reading Transformer Internals by Differencing Logit Lenses](https://mlsanity.com/paper/2609.09902/) (2026)
- [Affinity-Aware Sharding for Delayed Tensor Parallelism](https://mlsanity.com/paper/2609.13846/) (2026)
- [Structure is not mechanism: high-gain gated-FFN rows across text and genomic foundation models](https://mlsanity.com/paper/2609.17599/) (2026)
- [Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention](https://mlsanity.com/paper/2609.22005/) (2026)
- [Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction](https://mlsanity.com/paper/2609.31956/) (2026)
- [Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Models](https://mlsanity.com/paper/2609.32808/) (2026)
- [Which the Eye Fears: Writing with Read-Blindness Explains Massive Activations in Transformers](https://mlsanity.com/paper/2609.35630/) (2026)
- [Selecting The Most Informative Tokens in Natural Language Autoencoders](https://mlsanity.com/paper/2609.37040/) (2026)
- [Spectral Geometry of Attention: From Information Routing to Uncertainty](https://mlsanity.com/paper/2610.06012/) (2026)
- [Massive Activation Gating Channel in Large Language Models](https://mlsanity.com/paper/2610.07661/) (2026)

## Abstract (arXiv)

We study two recurring phenomena in Transformer language models: massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks, in which certain tokens attract disproportionate attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same tokens, but their functional roles and causal relationship remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally: they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters of the model. Attention sinks operate locally: they modulate attention outputs across heads and bias individual heads toward short-range dependencies. We identify the pre-norm configuration as the key choice that enables the co-occurrence, and show that ablating it causes the two phenomena to decouple.