ViM-VQ: Efficient Post-Training Vector Quantization for Visual Mamba

arXiv 2503.09509 ↗First posted 12 Mar 2025cs.CVcited by 3

On the map: State-space sequence models (S4, Mamba) → Vision Mamba backbones

Abstract (arXiv)

Visual Mamba networks (ViMs) extend the selective state space model (Mamba) to various vision tasks and demonstrate significant potential. As a promising compression technique, vector quantization (VQ) decomposes network weights into codebooks and assignments, significantly reducing memory usage and computational latency, thereby enabling the deployment of ViMs on edge devices. Although existing VQ methods have achieved extremely low-bit quantization (e.g., 3-bit, 2-bit, and 1-bit) in convolutional neural networks and Transformer-based networks, directly applying these methods to ViMs results in unsatisfactory accuracy. We identify several key challenges: 1) The weights of Mamba-based blocks in ViMs contain numerous outliers, significantly amplifying quantization errors. 2) When applied to ViMs, the latest VQ methods suffer from excessive memory consumption, lengthy calibration procedures, and suboptimal performance in the search for optimal codewords. In this paper, we propose ViM-VQ, an efficient post-training vector quantization method tailored for ViMs. ViM-VQ consists of two innovative components: 1) a fast convex combination optimization algorithm that efficiently updates both the convex combinations and the convex hulls to search for optimal codewords, and 2) an incremental vector quantization strategy that incrementally confirms optimal codewords to mitigate truncation errors. Experimental results demonstrate that ViM-VQ achieves state-of-the-art performance in low-bit quantization across various visual tasks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 43 of its 46 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

43 of 46 references.

Closest ancestors · 5

Same direction, newest first

Other references · 38

2017Attention Is All You Needcited by 59,932
2015Deep Residual Learning for Image Recognitioncited by 51,809
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
2024VMamba: Visual State Space Modelcited by 955
2019Learned Step Size Quantizationcited by 604

26 more

2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
2015An Introduction to Convolutional Neural Networkscited by 510
2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
2021Post-Training Quantization for Vision Transformercited by 274
2022Post-training Quantization on Diffusion Modelscited by 262
2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 211
2024Mamba-ND: Selective State Space Modeling for Multi-Dimensional Datacited by 84
2021Network Quantization with Element-wise Gradient Scalingcited by 82
2019And the Bit Goes Down: Revisiting the Quantization of Neural Networkscited by 78
2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 53
2016Towards the Limit of Network Quantizationcited by 49
2018Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutionscited by 33
2021DKM: Differentiable K-Means Clustering Layer for Neural Network Compressioncited by 31
2020Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networkscited by 28
2025MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methodscited by 26
2024VQ4DiT: Efficient Post-Training Vector Quantization for Diffusion Transformerscited by 12
2022Learning Low-Rank Representations for Model Compressioncited by 2
2023Network Memory Footprint Compression Through Jointly Learnable Codebooks and Mappingscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

2026ViM-Q: Scalable Algorithm-Hardware Co-Design for Vision Mamba Model Inference on FPGAcited by 0
2026Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Trainingcited by 0
2026Channel-Wise and Token-Aware Post-Training Quantization for Visual State Space Dualitycited by 0

Similar, not cited

Close in wording, no citation either way