mlsanityData to 14 Oct 2026

Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models

arXiv 2209.13325 ↗First posted 27 Sep 2022cs.LGcited by 181

Not on the map. Why?

Abstract (arXiv)

Transformer architecture has become the fundamental element of the widespread natural language processing~(NLP) models. With the trends of large NLP models, the increasing memory and computation costs hinder their efficient deployment on resource-limited devices. Therefore, transformer quantization attracts wide research interest. Recent work recognizes that structured outliers are the critical bottleneck for quantization performance. However, their proposed methods increase the computation overhead and still leave the outliers there. To fundamentally address this problem, this paper delves into the inherent inducement and importance of the outliers. We discover that boldsymbol γ in LayerNorm (LN) acts as a sinful amplifier for the outliers, and the importance of outliers varies greatly where some outliers provided by a few tokens cover a large area but can be clipped sharply without negative impacts. Motivated by these findings, we propose an outlier suppression framework including two components: Gamma Migration and Token-Wise Clipping. The Gamma Migration migrates the outlier amplifier to subsequent modules in an equivalent transformation, contributing to a more quantization-friendly model without any extra burden. The Token-Wise Clipping takes advantage of the large variance of token range and designs a token-wise coarse-to-fine pipeline, obtaining a clipping range with minimal final quantization loss in an efficient way. This framework effectively suppresses the outliers and can be used in a plug-and-play mode. Extensive experiments prove that our framework surpasses the existing works and, for the first time, pushes the 6-bit post-training BERT quantization to the full-precision (FP) level. Our code is available at https://github.com/wimh966/outliersuppression.

Key facts. Cited by 181 papers in the map (data through 14 Oct 2026). Builds on 44 of its 56 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

44 of 56 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  12. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
32 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  2. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  3. 2018Universal Transformerscited by 647
  4. 2019Learned Step Size Quantizationcited by 628
  5. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  6. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  7. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  8. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  9. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  10. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  11. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  12. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  13. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  14. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 193
  15. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  16. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  17. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  18. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  19. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  20. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  21. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  22. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  23. 2019Loss Aware Post-training Quantizationcited by 97
  24. 2019Towards Unified INT8 Training for Convolutional Neural Networkcited by 79
  25. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  26. 2020Robust Quantization: One Model to Rule Them Allcited by 63
  27. 2021Diversifying Sample Generation for Accurate Data-Free Quantizationcited by 63
  28. 2020EasyQuant: Post-training Quantization via Scale Optimizationcited by 57
  29. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  30. 2020Positional Artefacts Propagate Through Masked Language Model Embeddingscited by 35
  31. 2018Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inferencecited by 33
  32. 2020Once Quantization-Aware Training: High Performance Extremely Low-bit Architecture Searchcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 28; 2024: 74; 2025: 43; 2026: 33.

181 papers.

Sort:

By area:

  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  3. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  5. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  6. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  7. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  8. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  9. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  10. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  11. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  12. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
38 more
  1. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  2. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  3. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  4. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  5. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  6. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  7. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  8. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  9. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 100
  10. 2024Q-DiT: Accurate Post-Training Quantization for Diffusion Transformerscited by 94
  11. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  12. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 88
  13. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  14. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  15. 2024PTQ4DiT: Post-training Quantization for Diffusion Transformerscited by 79
  16. 2024Evaluating Quantized Large Language Modelscited by 77
  17. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  18. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 67
  19. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 64
  20. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  21. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 61
  22. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  23. 2023Training Transformers with 4-bit Integerscited by 60
  24. 2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 60
  25. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  26. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  27. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  28. 2024The Super Weight in Large Language Modelscited by 52
  29. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 50
  30. 2023Intriguing Properties of Quantization at Scalecited by 49
  31. 2024QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuningcited by 49
  32. 2024Q-VLM: Post-training Quantization for Large Vision-Language Modelscited by 47
  33. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 47
  34. 2024A Survey on Transformer Compressioncited by 46
  35. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 44
  36. 2024Low-Rank Quantization-Aware Training for LLMscited by 43
  37. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 41
  38. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38

50 most cited of 181 All 181 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking the Outlier Distribution in Large Language Models: An In-depth Study
  2. 2025Turning LLM Activations Quantization-Friendly
  3. 2025Adaptive Layer-Wise Transformations for Post-Training Quantization of Large Language Models
  4. 2026MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization
  5. 2026Astro: Activation-guided Structured Regularization for Outlier-Robust LLM Post-Training Quantization
  6. 2024DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation
  7. 2025From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction
  8. 2026Theory-optimal Quantization Based on Flatness