mlsanityData to 14 Oct 2026

Systematic Outliers in Large Language Models

arXiv 2502.06415 ↗First posted 10 Feb 2025cs.CL cs.AI cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Outliers have been widely observed in Large Language Models (LLMs), significantly impacting model performance and posing challenges for model compression. Understanding the functionality and formation mechanisms of these outliers is critically important. Existing works, however, largely focus on reducing the impact of outliers from an algorithmic perspective, lacking an in-depth investigation into their causes and roles. In this work, we provide a detailed analysis of the formation process, underlying causes, and functions of outliers in LLMs. We define and categorize three types of outliers-activation outliers, weight outliers, and attention outliers-and analyze their distributions across different dimensions, uncovering inherent connections between their occurrences and their ultimate influence on the attention mechanism. Based on these observations, we hypothesize and explore the mechanisms by which these outliers arise and function, demonstrating through theoretical derivations and experiments that they emerge due to the self-attention mechanism's softmax operation. These outliers act as implicit context-aware scaling factors within the attention mechanism. As these outliers stem from systematic influences, we term them systematic outliers. Our study not only enhances the understanding of Transformer-based LLMs but also shows that structurally eliminating outliers can accelerate convergence and improve model compression. The code is avilable at https://github.com/an-yongqi/systematic-outliers.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

25 of 33 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2023Mistral 7Bcited by 3,164
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2020GLU Variants Improve Transformercited by 1,804
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
13 more
  1. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  4. 2023The Falcon Series of Open Language Modelscited by 535
  5. 2024Massive Activations in Large Language Modelscited by 294
  6. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  7. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  8. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  9. 2024Theory, Analysis, and Best Practices for Sigmoid Self-Attentioncited by 64
  10. 2024Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularizationcited by 23
  11. 2024Unveiling Linguistic Regions in Large Language Modelscited by 22
  12. 2024Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMscited by 12
  13. 2024Is It a Free Lunch for Removing Outliers during Pretraining?cited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 21.

33 papers.

Sort:

By area:

  1. 2025Pretraining Large Language Models with NVFP4cited by 76
  2. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 35
  3. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 23
  4. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 23
  5. 2025Scaling Law for Quantization-Aware Trainingcited by 17
  6. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 17
  7. 2025Hidden Dynamics of Massive Activations in Transformer Trainingcited by 10
  8. 2025Towards Superior Quantization Accuracy: A Layer-sensitive Approachcited by 8
  9. 2025TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummiescited by 6
  10. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  11. 2026Revisiting [CLS] and Patch Token Interaction in Vision Transformerscited by 4
  12. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 3
21 more
  1. 2026What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMscited by 3
  2. 2026Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformerscited by 3
  3. 2025Energy-Efficient and Dequantization-Free Q-LLMs: A Spiking Neural Network Approach to Salient Value Mitigationcited by 2
  4. 2026Let ViT Speak: Generative Language-Image Pre-trainingcited by 2
  5. 2026Layer Collapse in Diffusion Language Modelscited by 2
  6. 2026Registers Matter for Pixel-Space Diffusion Transformerscited by 2
  7. 2026UniSVQ: 2-bit Unified Scalar-Vector Quantizationcited by 2
  8. 2026Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateauscited by 2
  9. 2025QWHA: Quantization-Aware Walsh-Hadamard Adaptation for Parameter-Efficient Fine-Tuning on Large Language Modelscited by 1
  10. 2025Neural Weight Compression for Language Modelscited by 1
  11. 2025Adaptive Layer-Wise Transformations for Post-Training Quantization of Large Language Modelscited by 1
  12. 2026HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioningcited by 1
  13. 2026Rethinking Adapter Placement: A Dominant Adaptation Module Perspectivecited by 1
  14. 2026Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMscited by 1
  15. 2026CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Expertscited by 0
  16. 2026Measuring Maximum Activations in Open Large Language Modelscited by 0
  17. 2026Vision Transformers Need Better Token Interactioncited by 0
  18. 2026Super Weights in LLMs and the Failure of Selective Trainingcited by 0
  19. 2026Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMscited by 0
  20. 2026Which the Eye Fears: Writing with Read-Blindness Explains Massive Activations in Transformerscited by 0
  21. 2026Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernelscited by 0