mlsanityData to 14 Oct 2026

Is It a Free Lunch for Removing Outliers during Pretraining?

arXiv 2402.12102 ↗First posted 19 Feb 2024cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

With the growing size of large language models, the role of quantization becomes increasingly significant. However, outliers present in weights or activations notably influence the performance of quantized models. Recently, citetqtransformer introduced a novel softmax function aimed at pretraining models in an outlier-free manner, thereby enhancing their suitability for quantization. Interestingly, we observed that such an approach leads to performance degradation in full precision. Building on this insight, we enhance the method by ensuring its normalization is invariant to sequence length, a crucial factor for bridging the gap between pretraining and fine-tuning. Moreover, this improved method also facilitates successful pretraining of causal language models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 16 of its 19 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

16 of 19 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  12. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
4 more
  1. 2019Revealing the Dark Secrets of BERTcited by 301
  2. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  3. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  4. 2023Intriguing Properties of Quantization at Scalecited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Systematic Outliers in Large Language Modelscited by 33
  2. 2026Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigationcited by 11

Similar, not cited

Close in wording, no citation either way

  1. 2024OutlierTune: Efficient Channel-Wise Quantization for Large Language Models
  2. 2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
  3. 2024Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
  4. 2023PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models
  5. 2024Outlier Reduction with Gated Attention for Improved Post-training Quantization in Large Sequence-to-sequence Speech Foundation Models
  6. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling
  7. 2023Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models
  8. 2026Theory-optimal Quantization Based on Flatness