# Q-S5: Towards Quantized State Space Models

arXiv: https://arxiv.org/abs/2406.09477 · HTML: https://mlsanity.com/paper/2406.09477/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 26 of 26)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://mlsanity.com/paper/2004.09602/) (2020)
- [Chaos as an interpretable benchmark for forecasting and data-driven modelling](https://mlsanity.com/paper/2110.05266/) (2021)
- [Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models](https://mlsanity.com/paper/2402.02244/) (2024)
- [A Survey on Methods and Theories of Quantized Neural Networks](https://mlsanity.com/paper/1808.04752/) (2018)
- [PokeBNN: A Binary Pursuit of Lightweight Accuracy](https://mlsanity.com/paper/2112.00133/) (2021)
- [Efficient Video and Audio processing with Loihi 2](https://mlsanity.com/paper/2310.03251/) (2023)
- [Pareto-Optimal Quantized ResNet Is Mostly 4-bit](https://mlsanity.com/paper/2105.03536/) (2021)
- [4-bit Conformer with Native Quantization Aware Training for Speech Recognition](https://mlsanity.com/paper/2203.15952/) (2022)
- [Binarized Neural Machine Translation](https://mlsanity.com/paper/2302.04907/) (2023)
- [Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs](https://mlsanity.com/paper/2310.02277/) (2023)
- [Recurrent Neural Networks With Limited Numerical Precision](https://mlsanity.com/paper/1608.06902/) (2016)
- [Sub-mW Neuromorphic SNN audio processing applications with Rockpool and Xylo](https://mlsanity.com/paper/2208.12991/) (2022)
- [Hardware Aware Training for Efficient Keyword Spotting on General Purpose and Specialized Hardware](https://mlsanity.com/paper/2009.04465/) (2020)
- [On the quantization of recurrent neural networks](https://mlsanity.com/paper/2101.05453/) (2021)

## Built on by (most cited first, 10 of 10)

- [Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2](https://mlsanity.com/paper/2503.18002/) (2025)
- [A Diagonal Structured State Space Model on Loihi 2 for Efficient Streaming Sequence Processing](https://mlsanity.com/paper/2409.15022/) (2024)
- [State-Space Model Inspired Multiple-Input Multiple-Output Spiking Neurons](https://mlsanity.com/paper/2504.02591/) (2025)
- [Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity](https://mlsanity.com/paper/2502.01330/) (2025)
- [Quantizing Small-Scale State-Space Models for Edge AI](https://mlsanity.com/paper/2506.12480/) (2025)
- [IMSSA: Deploying modern state-space models on memristive in-memory compute hardware](https://mlsanity.com/paper/2412.20215/) (2024)
- [Low-Bit Data Processing Using Multiple-Output Spiking Neurons with Non-linear Reset Feedback](https://mlsanity.com/paper/2508.06292/) (2025)
- [HadamRNN: Binary and Sparse Ternary Orthogonal RNNs](https://mlsanity.com/paper/2502.00047/) (2025)
- [QS4D: Quantization-aware training for efficient hardware deployment of structured state-space sequential models](https://mlsanity.com/paper/2507.06079/) (2025)
- [ADPTNet: Adaptive with Prescriptive Timescales Non-Linear SSM for Sequence Modelling](https://mlsanity.com/paper/2609.34034/) (2026)

## Abstract (arXiv)

In the quest for next-generation sequence modeling architectures, State Space Models (SSMs) have emerged as a potent alternative to transformers, particularly for their computational efficiency and suitability for dynamical systems. This paper investigates the effect of quantization on the S5 model to understand its impact on model performance and to facilitate its deployment to edge and resource-constrained platforms. Using quantization-aware training (QAT) and post-training quantization (PTQ), we systematically evaluate the quantization sensitivity of SSMs across different tasks like dynamical systems modeling, Sequential MNIST (sMNIST) and most of the Long Range Arena (LRA). We present fully quantized S5 models whose test accuracy drops less than 1% on sMNIST and most of the LRA. We find that performance on most tasks degrades significantly for recurrent weights below 8-bit precision, but that other components can be compressed further without significant loss of performance. Our results further show that PTQ only performs well on language-based LRA tasks whereas all others require QAT. Our investigation provides necessary insights for the continued development of efficient and hardware-optimized SSMs.