mlsanityData to 14 Oct 2026

Rethinking the Outlier Distribution in Large Language Models: An In-depth Study

arXiv 2505.21670 ↗First posted 27 May 2025cs.CL cs.AIcited by 4

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Investigating outliers in large language models (LLMs) is crucial due to their significant impact on various aspects of LLM performance, including quantization and compression. Outliers often cause considerable quantization errors, leading to degraded model performance. Identifying and addressing these outliers can enhance the accuracy and efficiency of the quantization process, enabling smoother deployment on edge devices or specialized hardware. Recent studies have identified two common types of outliers in LLMs: massive activations and channel-wise outliers. While numerous quantization algorithms have been proposed to mitigate their effects and maintain satisfactory accuracy, few have thoroughly explored the root causes of these outliers in depth. In this paper, we conduct a comprehensive investigation into the formation mechanisms of these outliers and propose potential strategies to mitigate their occurrence. Ultimately, we introduce some efficient approaches to eliminate most massive activations and channel-wise outliers with minimal impact on accuracy.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 34 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

27 of 34 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  10. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  12. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
15 more
  1. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  2. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  3. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  4. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  5. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  6. 2021Pay Attention to MLPscited by 368
  7. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  8. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  9. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  10. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 200
  11. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  12. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  13. 2023How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environmentcited by 56
  14. 2024Rome was Not Built in a Single Step: Hierarchical Prompting for LLM-based Chip Designcited by 24
  15. 2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspectivecited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Pretraining Large Language Models with NVFP4cited by 76
  2. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 3
  3. 2026Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardwarecited by 0
  4. 2026Understanding LLM Quantization through Activation-Guided Compensation and Orthogonal Residualscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024OutlierTune: Efficient Channel-Wise Quantization for Large Language Models
  2. 2025Turning LLM Activations Quantization-Friendly
  3. 2024Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs
  4. 2023Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models
  5. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
  6. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling
  7. 2026OffQ: Taming Structured Outliers in LLM Quantization by Offsetting
  8. 2024Mamba-PTQ: Outlier Channels in Recurrent Large Language Models