mlsanityData to 7 Oct 2026

CRVQ: Channel-Relaxed Vector Quantization for Extreme Compression of LLMs

arXiv 2412.09282 ↗First posted 12 Dec 2024cs.LG cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Powerful large language models (LLMs) are increasingly expected to be deployed with lower computational costs, enabling their capabilities on resource-constrained devices. Post-training quantization (PTQ) has emerged as a star approach to achieve this ambition, with best methods compressing weights to less than 2 bit on average. In this paper, we propose Channel-Relaxed Vector Quantization (CRVQ), a novel technique that significantly improves the performance of PTQ baselines at the cost of only minimal additional bits. This state-of-the-art extreme compression method achieves its results through two key innovations: (1) carefully selecting and reordering a very small subset of critical weight channels, and (2) leveraging extended codebooks to relax the constraint of critical channels. With our method, we demonstrate a 38.9% improvement over the current strongest sub-2-bit PTQ baseline, enabling nearer lossless 1-bit compression. Furthermore, our approach offers flexible customization of quantization bit-width and performance, providing a wider range of deployment options for diverse hardware platforms.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

35 of 39 references.

References · 35

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
23 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  6. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  7. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  11. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  12. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  13. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  14. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  15. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  16. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  17. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  18. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
  19. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  20. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  21. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
  22. 2024ARB-LLM: Alternating Refined Binarizations for Large Language Modelscited by 30
  23. 2024Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2025RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Modelscited by 2
  2. 2026ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUscited by 1
  3. 2026Fitting Is Not Enough: Smoothness in Extremely Quantized LLMscited by 1
  4. 2026HUOZIIME: An On-Device LLM-enhanced Input Method for Deep Personalizationcited by 0
  5. 2026OffQ: Taming Structured Outliers in LLM Quantization by Offsettingcited by 0
  6. 2026RSLM: Training-Free Vector Quantization for Approximate Nearest Neighbor Searchcited by 0
  7. 2026RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Modelscited by 0