# MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods

arXiv: https://arxiv.org/abs/2501.13484 · HTML: https://mlsanity.com/paper/2501.13484/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 23 Jan 2025.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://mlsanity.com/paper/2308.13137/) (2023)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [LocalMamba: Visual State Space Model with Windowed Selective Scan](https://mlsanity.com/paper/2403.09338/) (2024)
- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer](https://mlsanity.com/paper/2111.13824/) (2021)
- [RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers](https://mlsanity.com/paper/2212.08254/) (2022)
- [Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference](https://mlsanity.com/paper/2403.14520/) (2024)
- [Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data](https://mlsanity.com/paper/2402.05892/) (2024)
- [PD-Quant: Post-Training Quantization based on Prediction Difference Metric](https://mlsanity.com/paper/2212.07048/) (2022)
- [WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More](https://mlsanity.com/paper/2402.12065/) (2024)
- [A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset](https://mlsanity.com/paper/1806.00358/) (2018)
- [I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models](https://mlsanity.com/paper/2405.17849/) (2024)

## Built on by (most cited first, 29 of 29)

- [Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models](https://mlsanity.com/paper/2503.22879/) (2025)
- [CompTrack: Information Bottleneck-Guided Low-Rank Dynamic Token Compression for Point Cloud Tracking](https://mlsanity.com/paper/2511.15580/) (2025)
- [KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models](https://mlsanity.com/paper/2602.11184/) (2026)
- [FocusTrack: One-Stage Focus-and-Suppress Framework for 3D Point Cloud Object Tracking](https://mlsanity.com/paper/2602.24133/) (2026)
- [UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs](https://mlsanity.com/paper/2512.03383/) (2025)
- [On State Reduction in Linear Attention](https://mlsanity.com/paper/2602.04852/) (2026)
- [DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization](https://mlsanity.com/paper/2608.27513/) (2026)
- [Q-MambaIR: Accurate Quantized Mamba for Efficient Image Restoration](https://mlsanity.com/paper/2503.21970/) (2025)
- [VAEVQ: Enhancing Discrete Visual Tokenization through Variational Modeling](https://mlsanity.com/paper/2511.06863/) (2025)
- [Beyond Frame-wise Tracking: A Trajectory-based Paradigm for Efficient Point Cloud Tracking](https://mlsanity.com/paper/2509.11453/) (2025)
- [ViM-VQ: Efficient Post-Training Vector Quantization for Visual Mamba](https://mlsanity.com/paper/2503.09509/) (2025)
- [TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization](https://mlsanity.com/paper/2605.19561/) (2026)
- [RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Models](https://mlsanity.com/paper/2510.01240/) (2025)
- [SSDi8: Accurate and Efficient 8-bit Quantization for State Space Duality](https://mlsanity.com/paper/2608.21952/) (2026)
- [eMamba: Efficient Acceleration Framework for Mamba Models in Edge Computing](https://mlsanity.com/paper/2508.10370/) (2025)
- [FEMBA on the Edge: Physiologically-Aware Pre-Training, Quantization, and Deployment of a Bidirectional Mamba EEG Foundation Model on an Ultra-low Power Microcontroller](https://mlsanity.com/paper/2603.26716/) (2026)
- [MGVQ: Synergizing Multi-dimensional Sensitivity-Aware and Gradient-Hessian Fusion for Vector Quantization](https://mlsanity.com/paper/2605.24019/) (2026)
- [QuantFace: Efficient Quantization for Face Restoration](https://mlsanity.com/paper/2506.00820/) (2025)
- [Late Breaking Results: Quamba-SE: Soft-edge Quantizer for Activations in State Space Models](https://mlsanity.com/paper/2601.09451/) (2026)
- [COREY: Entropy-Guided Runtime Chunk Scheduling for Selective Scan Kernels](https://mlsanity.com/paper/2604.10597/) (2026)
- [ViM-Q: Scalable Algorithm-Hardware Co-Design for Vision Mamba Model Inference on FPGA](https://mlsanity.com/paper/2605.01935/) (2026)
- [Channel-Wise and Token-Aware Post-Training Quantization for Visual State Space Duality](https://mlsanity.com/paper/2609.16656/) (2026)
- [RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Models](https://mlsanity.com/paper/2609.25492/) (2026)
- [LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization](https://mlsanity.com/paper/2609.38166/) (2026)
- [STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization](https://mlsanity.com/paper/2609.38169/) (2026)
- [Low-Discrepancy Dither for Quantized Recurrent State Caches](https://mlsanity.com/paper/2609.39185/) (2026)
- [When Dynamic Scaling Fails: A Scale-State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMs](https://mlsanity.com/paper/2610.15139/) (2026)
- [Preserving Updates in Low-Bit Recurrent State Quantization](https://mlsanity.com/paper/2610.15709/) (2026)
- [Hardware-Friendly Token-Group Activation Quantization for Low-Bit Mamba Super-Resolution](https://mlsanity.com/paper/2610.16261/) (2026)

## Abstract (arXiv)

Mamba is an efficient sequence model that rivals Transformers and demonstrates significant potential as a foundational architecture for various tasks. Quantization is commonly used in neural networks to reduce model size and computational latency. However, applying quantization to Mamba remains underexplored, and existing quantization methods, which have been effective for CNN and Transformer models, appear inadequate for Mamba models (e.g., Quarot suffers a 21% accuracy drop on Vim-T^dagger even under W8A8). We have pioneered the exploration of this issue and identified several key challenges. First, significant outliers are present in gate projections, output projections, and matrix multiplications. Second, Mamba's unique parallel scan further amplifies these outliers, leading to uneven and heavy-tailed data distributions. Third, even with the application of the Hadamard transform, the variance across channels in weights and activations still remains inconsistent. To these ends, we propose MambaQuant, a post-training quantization (PTQ) framework consisting of: 1) Karhunen-Loeve Transformation (KLT) enhanced rotation, rendering the rotation matrix adaptable to diverse channel distributions. 2) Smooth-Fused rotation, which equalizes channel variances and can merge additional parameters into model weights. Experiments show that MambaQuant can quantize both weights and activations into 8-bit with less than 1% accuracy loss for Mamba-based vision and language tasks. To the best of our knowledge, MambaQuant is the first comprehensive PTQ design for the Mamba family, paving the way for further advancements in its application.