RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Models
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing tasks. However, their exponentially increasing parameters pose significant challenges for deployment on resource-constrained devices. Vector Quantization (VQ) shows great promise for low-bit quantization (e.g., 2 to 4 bits), but existing work faces two key challenges: unconstrained direction error and suboptimal bit allocation. In this paper, we propose RSAVQ, a novel VQ framework to enhance extremely low-bit quantization for LLMs. RSAVQ introduces two geometry-driven innovations that effectively mitigate above limitations: (1) Error Direction Sensitivity Guidance (EDSG), which leverages the Fisher Information Matrix (FIM)-induced Riemannian metric to project quantization errors onto low-sensitivity directions in the parameter space. Specifically, this projection is performed along the negative natural gradient direction, which effectively suppresses error expansion. (2) Weight Channel Sensitivity Guidance (WCSG) , which constructs a channel-wise sensitivity metric via FIM curvature analysis to dynamically guide bit resource allocation. The approach facilitates a globally optimal quantization solution within prescribed bit constraints. Experiments demonstrate that RSAVQ outperforms existing methods for LLMs. For example, in 2-bit quantization of LLaMA-3 8B, RSAVQ leads baselines like VPTQ and QuIP# by 0.4 in perplexity (PPL) and 1.5 in zero-shot accuracy. This work offers a practical solution for constrained environments and a theoretical bridge between information geometry and the quantization of neural networks, advancing efficient deep learning.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 24 Sep 2025.
Builds on
39 of 55 references.
References · 39
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2023Visual Instruction Tuningcited by 7,792
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2016Pruning Filters for Efficient ConvNetscited by 1,129
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
- 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
27 more
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
- 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
- 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
- 2014New insights and perspectives on the natural gradient methodcited by 435
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
- 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
- 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
- 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
- 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 94
- 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 86
- 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
- 2018A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Datasetcited by 54
- 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 53
- 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
- 2024DB-LLM: Accurate Dual-Binarization for Efficient LLMscited by 35
- 2025MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidancecited by 32
- 2018Fisher Information and Natural Gradient Learning of Random Deep Networkscited by 28
- 2025MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methodscited by 26
- 2023A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifoldcited by 21
- 2023Riemannian Residual Neural Networkscited by 21
- 2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Modelscited by 19
- 2022FIT: A Metric for Model Sensitivitycited by 10
- 2024CRVQ: Channel-Relaxed Vector Quantization for Extreme Compression of LLMscited by 7
- 2025PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decouplingcited by 5
- 2025RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantizationcited by 4
- 2024DRIVE: Dual Gradient-Based Rapid Iterative Pruningcited by 3
- 2021Probabilistic Learning Vector Quantization on Manifold of Symmetric Positive Definite Matricescited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.