mlsanityData to 7 Oct 2026

Massive Activations in Large Language Models

arXiv 2402.17762 ↗First posted 27 Feb 2024cs.CL cs.LGcited by 277

Not on the map. Why?

Cited by 25 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demonstrate the widespread existence of massive activations across various LLMs and characterize their locations. Second, we find their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs. Third, these massive activations lead to the concentration of attention probabilities to their corresponding tokens, and further, implicit bias terms in the self-attention output. Last, we also study massive activations in Vision Transformers. Code is available at https://github.com/locuslab/massive-activations.

Key facts. Cited by 277 papers in the map (data through 7 Oct 2026). Builds on 44 of its 55 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

44 of 55 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  12. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
32 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2023Mistral 7Bcited by 3,065
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2024Mixtral of Expertscited by 1,785
  13. 2019Root Mean Square Layer Normalizationcited by 1,425
  14. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  15. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  16. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  17. 2022In-context Learning and Induction Headscited by 930
  18. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  19. 2023Vision Transformers Need Registerscited by 844
  20. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  21. 2023Large Language Models as Optimizerscited by 830
  22. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  23. 2023The Falcon Series of Open Language Modelscited by 526
  24. 2019Revealing the Dark Secrets of BERTcited by 300
  25. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  26. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  27. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  28. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  29. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  30. 2023Intriguing Properties of Quantization at Scalecited by 48
  31. 2023Unveiling A Core Linguistic Region in Large Language Modelscited by 12
  32. 2023Role of Bias Terms in Dot-Product Attentioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 99; 2026: 141.

277 papers.

Sort:

By area:

  1. 2024DeepSeek-V3 Technical Reportcited by 724
  2. 2024Scaling and evaluating sparse autoencoderscited by 530
  3. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  5. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  6. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  7. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  8. 2026MiMo-V2-Flash Technical Reportcited by 163
  9. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  10. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  11. 2025Why do LLMs attend to the first token?cited by 122
  12. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
38 more
  1. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  2. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  3. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  4. 2025LongCat-Flash Technical Reportcited by 54
  5. 2024The Super Weight in Large Language Modelscited by 51
  6. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  7. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  8. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  9. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47
  10. 2024TinyFusion: Diffusion Transformers Learned Shallowcited by 44
  11. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 44
  12. 2024Low-Rank Quantization-Aware Training for LLMscited by 42
  13. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
  14. 2025Vision Transformers Don't Need Trained Registerscited by 39
  15. 2024Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Modelscited by 38
  16. 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 32
  17. 2025Systematic Outliers in Large Language Modelscited by 32
  18. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 32
  19. 2024Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscited by 31
  20. 2024Training Tensor Attention Efficiently: From Cubic to Almost Linear Timecited by 29
  21. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  22. 2026Qwen-Image-2.0 Technical Reportcited by 28
  23. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 26
  24. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 23
  25. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
  26. 2024Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMscited by 21
  27. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 21
  28. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 21
  29. 2025Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activationscited by 20
  30. 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 19
  31. 2024HSR-Enhanced Sparse Attention Accelerationcited by 19
  32. 2025Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinkingcited by 19
  33. 2025To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Modelscited by 18
  34. 2025Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddingscited by 16
  35. 2025FPTQuant: Function-Preserving Transforms for LLM Quantizationcited by 16
  36. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 16
  37. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  38. 2025Interpreting the Repeated Token Phenomenon in Large Language Modelscited by 15

50 most cited of 277 All 277 (JSON)