mlsanityData to 14 Oct 2026

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals

arXiv 2412.14363 ↗First posted 18 Dec 2024cs.LG cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Post-training quantization (PTQ) of large language models (LLMs) holds the promise in reducing the prohibitive computational cost at inference time. Quantization of all weight, activation and key-value (KV) cache tensors to 4-bit without significantly degrading generalizability is challenging, due to the high quantization error caused by extreme outliers in activations. To tackle this problem, we propose ResQ, a PTQ method that pushes further the state-of-the-art. By means of principal component analysis (PCA), it identifies a low-rank subspace (in practice 1/8 of the hidden dimension) in which activation variances are highest, and keep the coefficients within this subspace in high precision, e.g. 8-bit, while quantizing the rest to 4-bit. Within each subspace, invariant random rotation is applied to further suppress outliers. We show that this is a provably optimal mixed precision quantization scheme that minimizes error. With the Llama and Qwen2.5 families of models, we demonstrate that ResQ outperforms recent uniform and mixed precision PTQ methods on a variety of benchmarks, achieving up to 33% lower perplexity on Wikitext than the next best method SpinQuant, and upto 3× speedup over 16-bit baseline. Code is available at https://github.com/utkarsh-dmx/project-resq.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 52 of its 62 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

52 of 62 references.

References · 52

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
38 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  7. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  11. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  12. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  13. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  14. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  15. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  16. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  17. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  18. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  19. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  20. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  21. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  22. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  23. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  24. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  25. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  26. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  27. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  28. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  29. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  30. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 72
  31. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  32. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 64
  33. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 63
  34. 2023Training Transformers with 4-bit Integerscited by 60
  35. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Modelscited by 39
  36. 2024APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Modelscited by 34
  37. 2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projectioncited by 29
  38. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Modelscited by 22

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 25.

36 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  2. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 46
  3. 2026ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMscited by 15
  4. 2025KurTail : Kurtosis-based LLM Quantizationcited by 12
  5. 2025MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Modelscited by 11
  6. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correctioncited by 10
  7. 2026TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantizationcited by 8
  8. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  9. 2025TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummiescited by 6
  10. 2026LoopQ: Quantization for Looped Language Modelscited by 6
  11. 2026DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantizationcited by 6
  12. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 5
24 more
  1. 2026DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformerscited by 5
  2. 2025Universality of Layer-Level Entropy-Weighted Quantization Beyond Model Architecture and Sizecited by 4
  3. 2025A Comprehensive Evaluation on Quantization Techniques for Large Language Modelscited by 3
  4. 2025PAHQ: Accelerating Automated Circuit Discovery through Mixed-Precision Inference Optimizationcited by 3
  5. 2026dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formatscited by 3
  6. 2026MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantizationcited by 3
  7. 2026MF-QAT: Multi-Format Quantization-Aware Training for Elastic Inferencecited by 1
  8. 2026OASIS: Online Activation Subspace Learning for Memory-Efficient Trainingcited by 1
  9. 2026OffQ: Taming Structured Outliers in LLM Quantization by Offsettingcited by 1
  10. 2026Multi-Bitwidth Quantization for LLMs Using Additive Codebookscited by 1
  11. 2025Cat: Post-Training Quantization Error Reduction via Cluster-based Affine Transformationcited by 0
  12. 2026CRoPE: Efficient Parametrization of Rotary Positional Embeddingcited by 0
  13. 2026CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMscited by 0
  14. 2026Joint Structural Pruning and Mixed-Precision Quantization for LLM Compressioncited by 0
  15. 2026SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inferencecited by 0
  16. 2026GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inferencecited by 0
  17. 2026Language-Conditional Dequantization: Recovering What Quantization Steals from Non-English Languagescited by 0
  18. 2026TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Accelerationcited by 0
  19. 2026Component Type, Not Reconstruction Error, Predicts Attention Quantization Sensitivitycited by 0
  20. 2026PrismQuant: Optimal Null-Space Rotations for Grouped Quantizerscited by 0
  21. 2026JustQuant: You Don't Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantizationcited by 0
  22. 2026AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generationcited by 0
  23. 2026DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMscited by 0
  24. 2026LoopQuant: Accurate and Efficient Post-Training Quantization for Looped Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
  2. 2025PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models
  3. 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Models
  4. 2023CBQ: Cross-Block Quantization for Large Language Models
  5. 2024LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices
  6. 2025Rethinking Post-Training Quantization: Introducing a Statistical Pre-Calibration Approach
  7. 2025PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models
  8. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation