# Mamba-PTQ: Outlier Channels in Recurrent Large Language Models

arXiv: https://arxiv.org/abs/2407.12397 · HTML: https://mlsanity.com/paper/2407.12397/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models](https://mlsanity.com/paper/2402.19427/) (2024)
- [Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing](https://mlsanity.com/paper/2306.12929/) (2023)
- [Do Long-Range Language Models Actually Use Long-Range Context?](https://mlsanity.com/paper/2109.09115/) (2021)
- [RecurrentGemma: Moving Past Transformers for Efficient Open Language Models](https://mlsanity.com/paper/2404.07839/) (2024)
- [Efficient Video and Audio processing with Loihi 2](https://mlsanity.com/paper/2310.03251/) (2023)
- [Hardware Aware Training for Efficient Keyword Spotting on General Purpose and Specialized Hardware](https://mlsanity.com/paper/2009.04465/) (2020)
- [FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization](https://mlsanity.com/paper/2402.17985/) (2024)

## Built on by (most cited first, 15 of 15)

- [Quamba: A Post-Training Quantization Recipe for Selective State Space Models](https://mlsanity.com/paper/2410.13229/) (2024)
- [Bi-Mamba: Towards Accurate 1-Bit State Space Models](https://mlsanity.com/paper/2411.11843/) (2024)
- [LightMamba: Efficient Mamba Acceleration on FPGA with Quantization and Hardware Co-design](https://mlsanity.com/paper/2502.15260/) (2025)
- [Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2](https://mlsanity.com/paper/2503.18002/) (2025)
- [OuroMamba: A Data-Free Quantization Framework for Vision Mamba](https://mlsanity.com/paper/2503.10959/) (2025)
- [Post-Training Quantization for Vision Mamba with k-Scaled Quantization and Reparameterization](https://mlsanity.com/paper/2501.16738/) (2025)
- [Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity](https://mlsanity.com/paper/2502.01330/) (2025)
- [Quantizing Small-Scale State-Space Models for Edge AI](https://mlsanity.com/paper/2506.12480/) (2025)
- [A KL Lens on Quantization: Fast, Forward-Only Sensitivity for Mixed-Precision SSM-Transformer Models](https://mlsanity.com/paper/2604.13440/) (2026)
- [Neural Weight Compression for Language Models](https://mlsanity.com/paper/2510.11234/) (2025)
- [Decoder-Free Distillation for Quantized Image Restoration](https://mlsanity.com/paper/2603.09624/) (2026)
- [Ternary Mamba: Grouped Quantization-Aware Training of W1.58A16 State Space Models](https://mlsanity.com/paper/2606.18114/) (2026)
- [TinyCast: Probabilistic Zero-Shot Forecasting with Computed Periodicity](https://mlsanity.com/paper/2608.15767/) (2026)
- [Late Breaking Results: Quamba-SE: Soft-edge Quantizer for Activations in State Space Models](https://mlsanity.com/paper/2601.09451/) (2026)
- [ViM-Q: Scalable Algorithm-Hardware Co-Design for Vision Mamba Model Inference on FPGA](https://mlsanity.com/paper/2605.01935/) (2026)

## Abstract (arXiv)

Modern recurrent layers are emerging as a promising path toward edge deployment of foundation models, especially in the context of large language models (LLMs). Compressing the whole input sequence in a finite-dimensional representation enables recurrent layers to model long-range dependencies while maintaining a constant inference cost for each token and a fixed memory requirement. However, the practical deployment of LLMs in resource-limited environments often requires further model compression, such as quantization and pruning. While these techniques are well-established for attention-based models, their effects on recurrent layers remain underexplored. In this preliminary work, we focus on post-training quantization for recurrent LLMs and show that Mamba models exhibit the same pattern of outlier channels observed in attention-based LLMs. We show that the reason for the difficulty of quantizing SSMs is caused by activation outliers, similar to those observed in transformer-based LLMs. We report baseline results for post-training quantization of Mamba that do not take into account the activation outliers and suggest first steps for outlier-aware quantization.