# Extreme Compression of Large Language Models via Additive Quantization

arXiv: https://arxiv.org/abs/2401.06118 · HTML: https://mlsanity.com/paper/2401.06118/

## Key facts

- Cited by 232 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 66 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction](https://mlsanity.com/paper/2102.05426/) (2021)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [SqueezeLLM: Dense-and-Sparse Quantization](https://mlsanity.com/paper/2306.07629/) (2023)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)

## Built on by (most cited first, 30 of 232)

- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [EfficientQAT: Efficient Quantization-Aware Training for Large Language Models](https://mlsanity.com/paper/2407.11062/) (2024)
- [QTIP: Quantization with Trellises and Incoherence Processing](https://mlsanity.com/paper/2406.11235/) (2024)
- [GPTVQ: The Blessing of Dimensionality for LLM Quantization](https://mlsanity.com/paper/2402.15319/) (2024)
- [Exploiting LLM Quantization](https://mlsanity.com/paper/2405.18137/) (2024)
- [SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2405.14917/) (2024)
- [PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression](https://mlsanity.com/paper/2405.14852/) (2024)
- [Compressing Large Language Models using Low Rank and Low Precision Decomposition](https://mlsanity.com/paper/2405.18886/) (2024)
- [VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2409.17066/) (2024)
- [Training-Free Activation Sparsity in Large Language Models](https://mlsanity.com/paper/2408.14690/) (2024)
- [MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models](https://mlsanity.com/paper/2408.11743/) (2024)
- [ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization](https://mlsanity.com/paper/2502.02631/) (2025)
- [Mixture Compressor for Mixture-of-Experts LLMs Gains More](https://mlsanity.com/paper/2410.06270/) (2024)
- [Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization](https://mlsanity.com/paper/2504.09629/) (2025)
- [ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals](https://mlsanity.com/paper/2412.14363/) (2024)
- [LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit](https://mlsanity.com/paper/2405.06001/) (2024)
- [GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance](https://mlsanity.com/paper/2505.07004/) (2025)
- [Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization](https://mlsanity.com/paper/2509.23202/) (2025)
- [A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms](https://mlsanity.com/paper/2409.16694/) (2024)
- [Fast Matrix Multiplications for Lookup Table-Quantized LLMs](https://mlsanity.com/paper/2407.10960/) (2024)
- [Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models](https://mlsanity.com/paper/2501.19392/) (2025)
- [Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization](https://mlsanity.com/paper/2404.03605/) (2024)
- [Model-Preserving Adaptive Rounding](https://mlsanity.com/paper/2505.22988/) (2025)
- [LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid](https://mlsanity.com/paper/2407.10032/) (2024)
- [ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models](https://mlsanity.com/paper/2408.08554/) (2024)
- [Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale](https://mlsanity.com/paper/2407.12327/) (2024)

## Abstract (arXiv)

The emergence of accurate open large language models (LLMs) has led to a race towards performant quantization techniques which can enable their execution on end-user devices. In this paper, we revisit the problem of "extreme" LLM compression-defined as targeting extremely low bit counts, such as 2 to 3 bits per parameter-from the point of view of classic methods in Multi-Codebook Quantization (MCQ). Our algorithm, called AQLM, generalizes the classic Additive Quantization (AQ) approach for information retrieval to advance the state-of-the-art in LLM compression, via two innovations: 1) learned additive quantization of weight matrices in input-adaptive fashion, and 2) joint optimization of codebook parameters across each transformer blocks. Broadly, AQLM is the first scheme that is Pareto optimal in terms of accuracy-vs-model-size when compressing to less than 3 bits per parameter, and significantly improves upon all known schemes in the extreme compression (2bit) regime. In addition, AQLM is practical: we provide fast GPU and CPU implementations of AQLM for token generation, which enable us to match or outperform optimized FP16 implementations for speed, while executing in a much smaller memory footprint.