# Sparse Mamba: Introducing Controllability, Observability, And Stability To Structural State Space Models

arXiv: https://arxiv.org/abs/2409.00563 · HTML: https://mlsanity.com/paper/2409.00563/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 6 of its 17 references that are in the map.
- First posted on arXiv 31 Aug 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)

## Built on by (most cited first, 1 of 1)

- [Observability conditions for neural state-space models with eigenvalues and their roots of unity](https://mlsanity.com/paper/2504.15758/) (2025)

## Abstract (arXiv)

Structured state space models' (SSMs) development in recent studies, such as Mamba and Mamba2, outperformed and solved the computational inefficiency of transformers and large language models at small to medium scale. In this work, we introduce the concept of controllability and observability to the original Mamba SSM's architecture in our Sparse-Mamba (S-Mamba) for natural language processing (NLP) applications. Moreover, we reinforce stability on the nxn A matrix on Mmaba2. The Mamba SSMs architecture drops the need for attention layers or multilayer perception blocks in transformers. However, current Mamba models lack reinforcement of controllability in state-space equations for computing the A, B, C, and D matrices at each time step, leading to increased complexity and computational costs. Furthermore, the A matrix in Mamba2 is not always stable. We demonstrate a reduction of parameters compared to the first published Mamba and Mamba2. We showcase an improvement in perplexity by 5% and a decrease in training time by 3% after reinforcing controllability and observability on the original Mamba architecture in our proposed S-Mamba. We further enforce stability on the A matrix in Mamba2 to improve the loss and perplexity of the model. The controllable and stable n × n state matrix A is sparse, and it has only n free parameters. Our novel approach will ensure controllable/observable and stable SSMs, which will be the gate key for Mamba3.