mlsanityData to 14 Oct 2026

Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing

arXiv 2306.12929 ↗First posted 22 Jun 2023cs.LG cs.AI cs.CLcited by 151

Not on the map. Why?

Abstract (arXiv)

Transformer models have been widely adopted in various domains over the last years, and especially large language models have advanced the field of AI significantly. Due to their size, the capability of these networks has increased tremendously, but this has come at the cost of a significant increase in necessary compute. Quantization is one of the most effective ways to reduce the computational time and memory consumption of neural networks. Many studies have shown, however, that modern transformer models tend to learn strong outliers in their activations, making them difficult to quantize. To retain acceptable performance, the existence of these outliers requires activations to be in higher bitwidth or the use of different numeric formats, extra fine-tuning, or other workarounds. We show that strong outliers are related to very specific behavior of attention heads that try to learn a "no-op" or just a partial update of the residual. To achieve the exact zeros needed in the attention matrix for a no-update, the input to the softmax is pushed to be larger and larger during training, causing outliers in other parts of the network. Based on these observations, we propose two simple (independent) modifications to the attention mechanism - clipped softmax and gated attention. We empirically show that models pre-trained using our methods learn significantly smaller outliers while maintaining and sometimes even improving the floating-point task performance. This enables us to quantize transformers to full INT8 quantization of the activations without any additional effort. We demonstrate the effectiveness of our methods on both language models (BERT, OPT) and vision transformers.

Key facts. Cited by 151 papers in the map (data through 14 Oct 2026). Builds on 62 of its 80 references that are in the map. First posted on arXiv 22 Jun 2023.

Builds on

62 of 80 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2016Layer Normalizationcited by 5,669
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
38 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  2. 2014Going Deeper with Convolutionscited by 4,241
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  13. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  14. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  15. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  16. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  17. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  18. 2021Going deeper with Image Transformerscited by 764
  19. 2019Learned Step Size Quantizationcited by 628
  20. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  21. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  22. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  23. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  24. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  25. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  26. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  27. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  28. 2021A White Paper on Neural Network Quantizationcited by 472
  29. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  30. 2022DeiT III: Revenge of the ViTcited by 419
  31. 2015Deep Learning with Limited Numerical Precisioncited by 376
  32. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  33. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  34. 2019Revealing the Dark Secrets of BERTcited by 301
  35. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  36. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  37. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  38. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 49; 2025: 37; 2026: 55.

151 papers.

Sort:

By area:

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  3. 2024Massive Activations in Large Language Modelscited by 294
  4. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  5. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  6. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  7. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  8. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
  9. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  10. 2024Evaluating Quantized Large Language Modelscited by 77
  11. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 67
  12. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
38 more
  1. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
  2. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  3. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 52
  4. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 50
  5. 2024Low-Rank Quantization-Aware Training for LLMscited by 43
  6. 2024How Does Quantization Affect Multilingual LLMs?cited by 43
  7. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 41
  8. 2025Vision Transformers Don't Need Trained Registerscited by 41
  9. 2024Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Modelscited by 38
  10. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  11. 2024Activation Scaling for Steering and Interpreting Language Modelscited by 35
  12. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 35
  13. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 33
  14. 2024Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantizationcited by 33
  15. 2024Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscited by 31
  16. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  17. 2024Beat this! Accurate beat tracking without DBN postprocessingcited by 27
  18. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  19. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 27
  20. 2024What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbationcited by 23
  21. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 23
  22. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmaxcited by 22
  23. 2024Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMscited by 21
  24. 2023Training and inference of large language models using 8-bit floating pointcited by 20
  25. 2024StableMask: Refining Causal Masking in Decoder-only Transformercited by 20
  26. 2024Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Actioncited by 19
  27. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  28. 2024MobileQuant: Mobile-friendly Quantization for On-device Language Modelscited by 17
  29. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 17
  30. 2025Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddingscited by 16
  31. 2025Interpreting the Repeated Token Phenomenon in Large Language Modelscited by 16
  32. 2025FPTQuant: Function-Preserving Transforms for LLM Quantizationcited by 16
  33. 2023Softmax Bias Correction for Quantized Generative Modelscited by 15
  34. 2024Mamba-PTQ: Outlier Channels in Recurrent Large Language Modelscited by 15
  35. 2024RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantizationcited by 13
  36. 2024Differential Transformercited by 13
  37. 2024Methods of improving LLM training stabilitycited by 13
  38. 2024From Attention to Activation: Unravelling the Enigmas of Large Language Modelscited by 13

50 most cited of 151 All 151 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling
  2. 2026Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs
  3. 2024Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
  4. 2026Theory-optimal Quantization Based on Flatness
  5. 2024OutlierTune: Efficient Channel-Wise Quantization for Large Language Models
  6. 2026Pretraining Transformers with Quantized Softmax in Attention
  7. 2024Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview
  8. 2023Intriguing Properties of Quantization at Scale