# ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals

arXiv: https://arxiv.org/abs/2412.14363 · HTML: https://mlsanity.com/paper/2412.14363/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 62 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 30 of 52)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://mlsanity.com/paper/2308.13137/) (2023)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)

## Built on by (most cited first, 30 of 36)

- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations](https://mlsanity.com/paper/2501.16383/) (2025)
- [ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs](https://mlsanity.com/paper/2601.07475/) (2026)
- [KurTail : Kurtosis-based LLM Quantization](https://mlsanity.com/paper/2503.01483/) (2025)
- [MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Models](https://mlsanity.com/paper/2508.02343/) (2025)
- [KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction](https://mlsanity.com/paper/2510.05373/) (2025)
- [TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization](https://mlsanity.com/paper/2606.13054/) (2026)
- [Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs](https://mlsanity.com/paper/2604.04722/) (2026)
- [TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies](https://mlsanity.com/paper/2511.23225/) (2025)
- [LoopQ: Quantization for Looped Language Models](https://mlsanity.com/paper/2605.16343/) (2026)
- [DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization](https://mlsanity.com/paper/2608.27513/) (2026)
- [Mixed-Precision Quantization for Language Models: Techniques and Prospects](https://mlsanity.com/paper/2510.16805/) (2025)
- [DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers](https://mlsanity.com/paper/2605.16732/) (2026)
- [Universality of Layer-Level Entropy-Weighted Quantization Beyond Model Architecture and Size](https://mlsanity.com/paper/2503.04704/) (2025)
- [A Comprehensive Evaluation on Quantization Techniques for Large Language Models](https://mlsanity.com/paper/2507.17417/) (2025)
- [PAHQ: Accelerating Automated Circuit Discovery through Mixed-Precision Inference Optimization](https://mlsanity.com/paper/2510.23264/) (2025)
- [dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats](https://mlsanity.com/paper/2606.04115/) (2026)
- [MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization](https://mlsanity.com/paper/2606.15652/) (2026)
- [MF-QAT: Multi-Format Quantization-Aware Training for Elastic Inference](https://mlsanity.com/paper/2604.00529/) (2026)
- [OASIS: Online Activation Subspace Learning for Memory-Efficient Training](https://mlsanity.com/paper/2604.09406/) (2026)
- [OffQ: Taming Structured Outliers in LLM Quantization by Offsetting](https://mlsanity.com/paper/2606.07116/) (2026)
- [Multi-Bitwidth Quantization for LLMs Using Additive Codebooks](https://mlsanity.com/paper/2606.12876/) (2026)
- [Cat: Post-Training Quantization Error Reduction via Cluster-based Affine Transformation](https://mlsanity.com/paper/2509.26277/) (2025)
- [CRoPE: Efficient Parametrization of Rotary Positional Embedding](https://mlsanity.com/paper/2601.02728/) (2026)
- [CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs](https://mlsanity.com/paper/2604.26378/) (2026)
- [Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression](https://mlsanity.com/paper/2606.07819/) (2026)
- [SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference](https://mlsanity.com/paper/2606.26587/) (2026)
- [GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference](https://mlsanity.com/paper/2607.27694/) (2026)
- [Language-Conditional Dequantization: Recovering What Quantization Steals from Non-English Languages](https://mlsanity.com/paper/2608.11786/) (2026)
- [TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration](https://mlsanity.com/paper/2608.17336/) (2026)

## Abstract (arXiv)

Post-training quantization (PTQ) of large language models (LLMs) holds the promise in reducing the prohibitive computational cost at inference time. Quantization of all weight, activation and key-value (KV) cache tensors to 4-bit without significantly degrading generalizability is challenging, due to the high quantization error caused by extreme outliers in activations. To tackle this problem, we propose ResQ, a PTQ method that pushes further the state-of-the-art. By means of principal component analysis (PCA), it identifies a low-rank subspace (in practice 1/8 of the hidden dimension) in which activation variances are highest, and keep the coefficients within this subspace in high precision, e.g. 8-bit, while quantizing the rest to 4-bit. Within each subspace, invariant random rotation is applied to further suppress outliers. We show that this is a provably optimal mixed precision quantization scheme that minimizes error. With the Llama and Qwen2.5 families of models, we demonstrate that ResQ outperforms recent uniform and mixed precision PTQ methods on a variety of benchmarks, achieving up to 33% lower perplexity on Wikitext than the next best method SpinQuant, and upto 3× speedup over 16-bit baseline. Code is available at https://github.com/utkarsh-dmx/project-resq.