# VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models

arXiv: https://arxiv.org/abs/2409.17066 · HTML: https://mlsanity.com/paper/2409.17066/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 19 of 19)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks](https://mlsanity.com/paper/1911.03852/) (2019)
- [Extreme Compression of Large Language Models via Additive Quantization](https://mlsanity.com/paper/2401.06118/) (2024)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [Training with Quantization Noise for Extreme Model Compression](https://mlsanity.com/paper/2004.07320/) (2020)
- [OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models](https://mlsanity.com/paper/2306.02272/) (2023)
- [GPTVQ: The Blessing of Dimensionality for LLM Quantization](https://mlsanity.com/paper/2402.15319/) (2024)
- [And the Bit Goes Down: Revisiting the Quantization of Neural Networks](https://mlsanity.com/paper/1907.05686/) (2019)
- [Differentiable Product Quantization for End-to-End Embedding Compression](https://mlsanity.com/paper/1908.09756/) (2019)
- [DKM: Differentiable K-Means Clustering Layer for Neural Network Compression](https://mlsanity.com/paper/2108.12659/) (2021)

## Built on by (most cited first, 30 of 57)

- [Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization](https://mlsanity.com/paper/2504.09629/) (2025)
- [Block Rotation is All You Need for MXFP4 Quantization](https://mlsanity.com/paper/2511.04214/) (2025)
- [Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling](https://mlsanity.com/paper/2509.20241/) (2025)
- [Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis](https://mlsanity.com/paper/2502.13178/) (2025)
- [Channel-Wise Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2410.13056/) (2024)
- [ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transforms](https://mlsanity.com/paper/2509.09679/) (2025)
- [KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models](https://mlsanity.com/paper/2602.11184/) (2026)
- [NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache](https://mlsanity.com/paper/2505.18231/) (2025)
- [PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decoupling](https://mlsanity.com/paper/2506.05432/) (2025)
- [Fine-tuning Quantized Neural Networks with Zeroth-order Optimization](https://mlsanity.com/paper/2505.13430/) (2025)
- [DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration](https://mlsanity.com/paper/2505.23049/) (2025)
- [Pyramid Vector Quantization for LLMs](https://mlsanity.com/paper/2410.16926/) (2024)
- [QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decomposition](https://mlsanity.com/paper/2503.19353/) (2025)
- [RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization](https://mlsanity.com/paper/2505.03803/) (2025)
- [BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook](https://mlsanity.com/paper/2506.12040/) (2025)
- [LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs](https://mlsanity.com/paper/2511.06174/) (2025)
- [ViM-VQ: Efficient Post-Training Vector Quantization for Visual Mamba](https://mlsanity.com/paper/2503.09509/) (2025)
- [CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs](https://mlsanity.com/paper/2512.17970/) (2025)
- [ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generation](https://mlsanity.com/paper/2601.09703/) (2026)
- [OneComp: One-Line Revolution for Generative AI Model Compression](https://mlsanity.com/paper/2603.28845/) (2026)
- [RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Models](https://mlsanity.com/paper/2510.01240/) (2025)
- [MetaCluster: Enabling Deep Compression of Kolmogorov-Arnold Network](https://mlsanity.com/paper/2510.19105/) (2025)
- [D2Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs](https://mlsanity.com/paper/2602.02546/) (2026)
- [BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models](https://mlsanity.com/paper/2602.04163/) (2026)
- [EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation](https://mlsanity.com/paper/2605.04062/) (2026)
- [EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture](https://mlsanity.com/paper/2605.24144/) (2026)
- [JacQuant: STE-Free Quantization-Aware Training via Learned Jacobian Surrogates](https://mlsanity.com/paper/2605.25469/) (2026)
- [LiftQuant: Continuous Bit-Width LLM via Dimensional Lifting and Projection](https://mlsanity.com/paper/2606.04050/) (2026)
- [NoWag: A Unified Framework for Shape Preserving Compression of Large Language Models](https://mlsanity.com/paper/2504.14569/) (2025)
- [NeUQI: Near-Optimal Uniform Quantization Parameter Initialization for Low-Bit LLMs](https://mlsanity.com/paper/2505.17595/) (2025)

## Abstract (arXiv)

Scaling model size significantly challenges the deployment and inference of Large Language Models (LLMs). Due to the redundancy in LLM weights, recent research has focused on pushing weight-only quantization to extremely low-bit (even down to 2 bits). It reduces memory requirements, optimizes storage costs, and decreases memory bandwidth needs during inference. However, due to numerical representation limitations, traditional scalar-based weight quantization struggles to achieve such extreme low-bit. Recent research on Vector Quantization (VQ) for LLMs has demonstrated the potential for extremely low-bit model quantization by compressing vectors into indices using lookup tables. In this paper, we introduce Vector Post-Training Quantization (VPTQ) for extremely low-bit quantization of LLMs. We use Second-Order Optimization to formulate the LLM VQ problem and guide our quantization algorithm design by solving the optimization. We further refine the weights using Channel-Independent Second-Order Optimization for a granular VQ. In addition, by decomposing the optimization problem, we propose a brief and effective codebook initialization algorithm. We also extend VPTQ to support residual and outlier quantization, which enhances model accuracy and further compresses the model. Our experimental results show that VPTQ reduces model quantization perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on Mistral-7B, 4.41-7.34 on LLaMA-3 over SOTA at 2-bit, with an average accuracy improvement of 0.79-1.5% on LLaMA-2, 1% on Mistral-7B, 11-22% on LLaMA-3 on QA tasks on average. We only utilize 10.4-18.6% of the quantization algorithm execution time, resulting in a 1.6-1.8× increase in inference throughput compared to SOTA.