# Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization

arXiv: https://arxiv.org/abs/2404.03605 · HTML: https://mlsanity.com/paper/2404.03605/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 22 of 22)

- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://mlsanity.com/paper/2308.13137/) (2023)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [Extreme Compression of Large Language Models via Additive Quantization](https://mlsanity.com/paper/2401.06118/) (2024)
- [LSQ+: Improving low-bit quantization through learnable offsets and better initialization](https://mlsanity.com/paper/2004.09576/) (2020)
- [Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models](https://mlsanity.com/paper/2209.13325/) (2022)
- [Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss](https://mlsanity.com/paper/1808.05779/) (2018)
- [RPTQ: Reorder-based Post-training Quantization for Large Language Models](https://mlsanity.com/paper/2304.01089/) (2023)
- [BERT Busters: Outlier Dimensions that Disrupt Transformers](https://mlsanity.com/paper/2105.06990/) (2021)
- [Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks](https://mlsanity.com/paper/1903.08066/) (2019)
- [Robust Quantization: One Model to Rule Them All](https://mlsanity.com/paper/2002.07686/) (2020)
- [Outliers Dimensions that Disrupt Transformers Are Driven by Frequency](https://mlsanity.com/paper/2205.11380/) (2022)
- [QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models](https://mlsanity.com/paper/2310.09259/) (2023)
- [FP8 versus INT8 for efficient deep learning inference](https://mlsanity.com/paper/2303.17951/) (2023)
- [Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases](https://mlsanity.com/paper/2301.12017/) (2023)

## Built on by (most cited first, 23 of 23)

- [SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2405.14917/) (2024)
- [Training-Free Activation Sparsity in Large Language Models](https://mlsanity.com/paper/2408.14690/) (2024)
- [Mixture Compressor for Mixture-of-Experts LLMs Gains More](https://mlsanity.com/paper/2410.06270/) (2024)
- [Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling](https://mlsanity.com/paper/2512.02010/) (2025)
- [Systematic Outliers in Large Language Models](https://mlsanity.com/paper/2502.06415/) (2025)
- [Fast Matrix Multiplications for Lookup Table-Quantized LLMs](https://mlsanity.com/paper/2407.10960/) (2024)
- [The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks](https://mlsanity.com/paper/2603.05498/) (2026)
- [QuEST: Stable Training of LLMs with 1-Bit Weights and Activations](https://mlsanity.com/paper/2502.05003/) (2025)
- [Scaling Law for Quantization-Aware Training](https://mlsanity.com/paper/2505.14302/) (2025)
- [Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings](https://mlsanity.com/paper/2502.00919/) (2025)
- [HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs](https://mlsanity.com/paper/2501.02625/) (2025)
- [Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models](https://mlsanity.com/paper/2506.19697/) (2025)
- [Understanding and Minimising Outlier Features in Neural Network Training](https://mlsanity.com/paper/2405.19279/) (2024)
- [Hidden Dynamics of Massive Activations in Transformer Training](https://mlsanity.com/paper/2508.03616/) (2025)
- [Understanding and Steering the Cognitive Behaviors of Reasoning Models at Test-Time](https://mlsanity.com/paper/2512.24574/) (2025)
- [Beyond Outliers: A Study of Optimizers Under Quantization](https://mlsanity.com/paper/2509.23500/) (2025)
- [The Quantization Benefits of Residual-Free Transformers](https://mlsanity.com/paper/2605.25880/) (2026)
- [ReALLM: A general framework for LLM compression and fine-tuning](https://mlsanity.com/paper/2405.13155/) (2024)
- [Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs](https://mlsanity.com/paper/2604.07888/) (2026)
- [Energy-Efficient and Dequantization-Free Q-LLMs: A Spiking Neural Network Approach to Salient Value Mitigation](https://mlsanity.com/paper/2510.19498/) (2025)
- [Leveraging Second-Order Curvature for Efficient Learned Image Compression: Theory and Empirical Evidence](https://mlsanity.com/paper/2601.20769/) (2026)
- [Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decay](https://mlsanity.com/paper/2605.01330/) (2026)
- [Gradual Binary Search and Dimension Expansion : A general method for activation quantization in LLMs](https://mlsanity.com/paper/2504.13989/) (2025)

## Abstract (arXiv)

We consider the problem of accurate quantization for language models, where both the weights and activations are uniformly quantized to 4 bits per parameter, the lowest bitwidth format natively supported by GPU hardware. In this context, the key challenge is activation quantization: it is known that language models contain outlier channels whose values on average are orders of magnitude higher than than other channels, which prevents accurate low-bitwidth quantization with known techniques. We systematically study this phenomena and find that these outlier channels emerge early in training, and that they occur more frequently in layers with residual streams. We then propose a simple strategy which regularizes a layer's inputs via quantization-aware training (QAT) and its outputs via activation kurtosis regularization. We show that regularizing both the inputs and outputs is crucial for preventing a model's "migrating" the difficulty in input quantization to the weights, which makes post-training quantization (PTQ) of weights more difficult. When combined with weight PTQ, we show that our approach can obtain a W4A4 model that performs competitively to the standard-precision W16A16 baseline.