# Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models

arXiv: https://arxiv.org/abs/2409.18654 · HTML: https://mlsanity.com/paper/2409.18654/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 27 Sep 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Speech and audio SSMs](https://mlsanity.com/ssm/speech-and-audio-ssms/)

## Builds on (most cited first, 30 of 30)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [VM-UNet: Vision Mamba UNet for Medical Image Segmentation](https://mlsanity.com/paper/2402.02491/) (2024)
- [Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining](https://mlsanity.com/paper/2402.03302/) (2024)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Mamba in Speech: Towards an Alternative to Self-Attention](https://mlsanity.com/paper/2405.12609/) (2024)
- [The Microsoft 2017 Conversational Speech Recognition System](https://mlsanity.com/paper/1708.06073/) (2017)
- [Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM](https://mlsanity.com/paper/1706.02737/) (2017)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)
- [U-shaped Vision Mamba for Single Image Dehazing](https://mlsanity.com/paper/2402.04139/) (2024)
- [Very Deep Convolutional Networks for End-to-End Speech Recognition](https://mlsanity.com/paper/1610.03022/) (2016)
- [Weak-Mamba-UNet: Visual Mamba Makes CNN and ViT Work Better for Scribble-based Medical Image Segmentation](https://mlsanity.com/paper/2402.10887/) (2024)
- [Simplified Self-Attention for Transformer-based End-to-End Speech Recognition](https://mlsanity.com/paper/2005.10463/) (2020)
- [Exploring the Capability of Mamba in Speech Applications](https://mlsanity.com/paper/2406.16808/) (2024)
- [PoLyScriber: Integrated Fine-tuning of Extractor and Lyrics Transcriber for Polyphonic Music](https://mlsanity.com/paper/2207.07336/) (2022)
- [Self-Transriber: Few-shot Lyrics Transcription with Self-training](https://mlsanity.com/paper/2211.10152/) (2022)
- [Music-robust Automatic Lyrics Transcription of Polyphonic Music](https://mlsanity.com/paper/2204.03306/) (2022)

## Built on by (most cited first, 9 of 9)

- [Long-Form Speech Generation with Spoken Language Models](https://mlsanity.com/paper/2412.18603/) (2024)
- [Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative](https://mlsanity.com/paper/2508.09294/) (2025)
- [An Exploration of Mamba for Speech Self-Supervised Models](https://mlsanity.com/paper/2506.12606/) (2025)
- [MLMA: Towards Multilingual ASR With Mamba-based Architectures](https://mlsanity.com/paper/2510.18684/) (2025)
- [MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesis](https://mlsanity.com/paper/2603.12342/) (2026)
- [Keyword Mamba: Spoken Keyword Spotting with State Space Models](https://mlsanity.com/paper/2508.07363/) (2025)
- [An overview of neural architectures for self-supervised audio representation learning from masked spectrograms](https://mlsanity.com/paper/2509.18691/) (2025)
- [MORE: Multi-Objective Adversarial Attacks on Speech Recognition](https://mlsanity.com/paper/2601.01852/) (2026)
- [Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition](https://mlsanity.com/paper/2606.01135/) (2026)

## Abstract (arXiv)

Current automatic speech recognition systems struggle with modeling long speech sequences due to high quadratic complexity of Transformer-based models. Selective state space models such as Mamba has performed well on long-sequence modeling in natural language processing and computer vision tasks. However, research endeavors in speech technology tasks has been under-explored. We propose Speech-Mamba, which incorporates selective state space modeling in Transformer neural architectures. Long sequence representations with selective state space models in Speech-Mamba is complemented with lower-level representations from Transformer-based modeling. Speech-mamba achieves better capacity to model long-range dependencies, as it scales near-linearly with sequence length.