mlsanityData to 7 Oct 2026

Rethinking Mamba in Speech Processing by Self-Supervised Models

arXiv 2409.07273 ↗First posted 11 Sep 2024eess.AScited by 10

On the map: State-space sequence models (S4, Mamba) → Speech and audio SSMs

Abstract (arXiv)

The Mamba-based model has demonstrated outstanding performance across tasks in computer vision, natural language processing, and speech processing. However, in the realm of speech processing, the Mamba-based model's performance varies across different tasks. For instance, in tasks such as speech enhancement and spectrum reconstruction, the Mamba model performs well when used independently. However, for tasks like speech recognition, additional modules are required to surpass the performance of attention-based models. We propose the hypothesis that the Mamba-based model excels in "reconstruction" tasks within speech processing. However, for "classification tasks" such as Speech Recognition, additional modules are necessary to accomplish the "reconstruction" step. To validate our hypothesis, we analyze the previous Mamba-based Speech Models from an information theory perspective. Furthermore, we leveraged the properties of HuBERT in our study. We trained a Mamba-based HuBERT model, and the mutual information patterns, along with the model's performance metrics, confirmed our assumptions.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 28 of its 36 references that are in the map. First posted on arXiv 11 Sep 2024.

Builds on

28 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024SELD-Mamba: Selective State-Space Model for Sound Event Localization and Detection with Source Distance Estimation9 Aug
  2. 2024Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis13 Jul
  3. 2024Mamba in Speech: Towards an Alternative to Self-Attention21 May
  4. 2024SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model20 May
  5. 2024An Investigation of Incorporating Mamba for Speech Enhancement10 May

Other references · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  9. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  10. 2018MINE: Mutual Information Neural Estimationcited by 680
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  12. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
11 more
  1. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  2. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  3. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  4. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  5. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  6. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  7. 2024Is Mamba Capable of In-Context Learning?cited by 45
  8. 2023Enhancing Code-switching Speech Recognition with Interactive Language Biasescited by 20
  9. 2024When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detectioncited by 16
  10. 2024Aligning Speech to Languages to Enhance Code-switching Speech Recognitioncited by 13
  11. 2023Time-domain Speech Enhancement Assisted by Multi-resolution Frequency Encoder and Decodercited by 3

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 8.

10 papers.

  1. 2024Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decodingcited by 7
  2. 2025Deep Active Speech Cancellation with Mamba-Masking Networkcited by 2
  3. 2025An Exploration of Mamba for Speech Self-Supervised Modelscited by 2
  4. 2025Learning Human Motion with Temporally Conditional Mambacited by 2
  5. 2024Selective State Space Model for Monaural Speech Enhancementcited by 1
  6. 2025MLMA: Towards Multilingual ASR With Mamba-based Architecturescited by 1
  7. 2025Active Speech Enhancement: Active Speech Denoising Decliping and Deveraberationcited by 0
  8. 2025Keyword Mamba: Spoken Keyword Spotting with State Space Modelscited by 0
  9. 2025D2-Mamba: Dual-Scale Fusion and Dual-Path Scanning with SSMs for Shadow Removalcited by 0
  10. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrogramscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Omni-directional attention mechanism based on Mamba for speech separation
  2. 2024Mamba-SEUNet: Mamba UNet for Monaural Speech Enhancement
  3. 2024Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models
  4. 2024Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognition
  5. 2024The Hidden Attention of Mamba Models
  6. 2026From Monolingual to Multilingual: Evaluating Mamba for ASR in South African Languages
  7. 2024Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations
  8. 2024Leveraging Joint Spectral and Spatial Learning with MAMBA for Multichannel Speech Enhancement