# PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression

arXiv: https://arxiv.org/abs/2405.14852 · HTML: https://mlsanity.com/paper/2405.14852/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 85 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 30 of 48)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://mlsanity.com/paper/2308.13137/) (2023)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [SqueezeLLM: Dense-and-Sparse Quantization](https://mlsanity.com/paper/2306.07629/) (2023)

## Built on by (most cited first, 30 of 61)

- [QTIP: Quantization with Trellises and Incoherence Processing](https://mlsanity.com/paper/2406.11235/) (2024)
- [GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration](https://mlsanity.com/paper/2504.02692/) (2025)
- [UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making](https://mlsanity.com/paper/2506.17419/) (2025)
- [Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization](https://mlsanity.com/paper/2504.09629/) (2025)
- [GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance](https://mlsanity.com/paper/2505.07004/) (2025)
- [Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models](https://mlsanity.com/paper/2501.19392/) (2025)
- [Model-Preserving Adaptive Rounding](https://mlsanity.com/paper/2505.22988/) (2025)
- [Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale](https://mlsanity.com/paper/2407.12327/) (2024)
- [MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design](https://mlsanity.com/paper/2412.14590/) (2024)
- [QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2502.12346/) (2025)
- [Inference Optimization of Foundation Models on AI Accelerators](https://mlsanity.com/paper/2407.09111/) (2024)
- [ICQuant: Index Coding enables Low-bit LLM Quantization](https://mlsanity.com/paper/2505.00850/) (2025)
- [Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs](https://mlsanity.com/paper/2508.19432/) (2025)
- [Thanos: A Block-wise Pruning Algorithm for Efficient Large Language Model Compression](https://mlsanity.com/paper/2504.05346/) (2025)
- [Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression](https://mlsanity.com/paper/2504.07389/) (2025)
- [WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization](https://mlsanity.com/paper/2603.04956/) (2026)
- [Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization](https://mlsanity.com/paper/2409.18850/) (2024)
- [LPCD: Unified Framework from Layer-Wise to Submodule Quantization](https://mlsanity.com/paper/2512.01546/) (2025)
- [GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling](https://mlsanity.com/paper/2604.18556/) (2026)
- [Improving the Straight-Through Estimator with Zeroth-Order Information](https://mlsanity.com/paper/2510.23926/) (2025)
- [HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization](https://mlsanity.com/paper/2605.29843/) (2026)
- [CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs](https://mlsanity.com/paper/2507.07145/) (2025)
- [Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression](https://mlsanity.com/paper/2510.20984/) (2025)
- [Symmetric Pruning of Large Language Models](https://mlsanity.com/paper/2501.18980/) (2025)
- [RSQ: Learning from Important Tokens Leads to Better Quantized LLMs](https://mlsanity.com/paper/2503.01820/) (2025)
- [Addition is almost all you need: Compressing large language models with double binary factorization](https://mlsanity.com/paper/2505.11076/) (2025)
- [LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models](https://mlsanity.com/paper/2509.23729/) (2025)
- [CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs](https://mlsanity.com/paper/2512.17970/) (2025)
- [More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization](https://mlsanity.com/paper/2512.24545/) (2025)
- [OneComp: One-Line Revolution for Generative AI Model Compression](https://mlsanity.com/paper/2603.28845/) (2026)

## Abstract (arXiv)

There has been significant interest in "extreme" compression of large language models (LLMs), i.e., to 1-2 bits per parameter, which allows such models to be executed efficiently on resource-constrained devices. Existing work focused on improved one-shot quantization techniques and weight representations; yet, purely post-training approaches are reaching diminishing returns in terms of the accuracy-vs-bit-width trade-off. State-of-the-art quantization methods such as QuIP# and AQLM include fine-tuning (part of) the compressed parameters over a limited amount of calibration data; however, such fine-tuning techniques over compressed weights often make exclusive use of straight-through estimators (STE), whose performance is not well-understood in this setting. In this work, we question the use of STE for extreme LLM compression, showing that it can be sub-optimal, and perform a systematic study of quantization-aware fine-tuning strategies for LLMs. We propose PV-Tuning - a representation-agnostic framework that generalizes and improves upon existing fine-tuning strategies, and provides convergence guarantees in restricted cases. On the practical side, when used for 1-2 bit vector quantization, PV-Tuning outperforms prior techniques for highly-performant models such as Llama and Mistral. Using PV-Tuning, we achieve the first Pareto-optimal quantization for Llama 2 family models at 2 bits per parameter.