mlsanityData to 14 Oct 2026

Mamba in Speech: Towards an Alternative to Self-Attention

arXiv 2405.12609 ↗First posted 21 May 2024eess.AS cs.SDcited by 60

On the map: State-space sequence models (S4, Mamba) → Speech and audio SSMs

Abstract (arXiv)

Transformer and its derivatives have achieved success in diverse tasks across computer vision, natural language processing, and speech processing. To reduce the complexity of computations within the multi-head self-attention mechanism in Transformer, Selective State Space Models (i.e., Mamba) were proposed as an alternative. Mamba exhibited its effectiveness in natural language processing and computer vision tasks, but its superiority has rarely been investigated in speech signal processing. This paper explores solutions for applying Mamba to speech processing by discussing two typical speech processing tasks: speech recognition, which requires semantic and sequential information, and speech enhancement, which focuses primarily on sequential patterns. The experimental results confirm that bidirectional Mamba (BiMamba) consistently outperforms vanilla Mamba, highlighting the advantages of a bidirectional design for speech processing. Moreover, experiments demonstrate the effectiveness of BiMamba as an alternative to the self-attention module in the Transformer model and its derivates, particularly for the semantic-aware task. The crucial technologies for transferring Mamba to speech are then summarized in ablation studies and the discussion section, offering insights for extending this research to a broader scope of tasks.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 54 of its 102 references that are in the map. First posted on arXiv 21 May 2024.

Builds on

54 of 102 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024An Investigation of Incorporating Mamba for Speech Enhancement10 May
  2. 2024Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation27 Mar

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  12. 2021ViViT: A Video Vision Transformercited by 1,620
38 more
  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  2. 2021GPT Understands, Toocited by 716
  3. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  4. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  5. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  6. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  7. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  8. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  9. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  10. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  11. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Modelscited by 197
  12. 2020Real Time Speech Enhancement in the Waveform Domaincited by 186
  13. 2017SEGAN: Speech Enhancement Generative Adversarial Networkcited by 166
  14. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 162
  15. 2020DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancementcited by 141
  16. 2021MetricGAN+: An Improved Version of MetricGAN for Speech Enhancementcited by 114
  17. 2022StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberationcited by 105
  18. 2019MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancementcited by 104
  19. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  20. 2023MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectracited by 52
  21. 2019PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Networkcited by 51
  22. 2020The ASRU 2019 Mandarin-English Code-Switching Speech Recognition Challenge: Open Datasets, Tracks, Methods and Resultscited by 44
  23. 2020HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networkscited by 40
  24. 2019T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancementcited by 34
  25. 2018On the End-to-End Solution to Mandarin-English Code-switching Speech Recognitioncited by 30
  26. 2022Reducing Language confusion for Code-switching Speech Recognition with Token-level Language Diarizationcited by 29
  27. 2020Speech Enhancement using Self-Adaptation and Multi-Head Self-Attentioncited by 28
  28. 2024Multichannel Long-Term Streaming Neural Speech Enhancement for Static and Moving Speakerscited by 28
  29. 2021DCCRN+: Channel-wise Subband DCCRN with SNR Estimation for Speech Enhancementcited by 23
  30. 2022CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancementcited by 23
  31. 2020Improving GANs for Speech Enhancementcited by 22
  32. 2023Enhancing Code-switching Speech Recognition with Interactive Language Biasescited by 21
  33. 2021S-DCCRN: Super Wide Band DCCRN with learnable complex feature for speech enhancementcited by 14
  34. 2024Aligning Speech to Languages to Enhance Code-switching Speech Recognitioncited by 14
  35. 2022Speech Denoising in the Waveform Domain with Self-Attentioncited by 13
  36. 2023Explicit Estimation of Magnitude and Phase Spectra in Parallel for High-Quality Speech Enhancementcited by 13
  37. 2020WaveCRN: An Efficient Convolutional Recurrent Neural Network for End-to-end Speech Enhancementcited by 12
  38. 2024An Empirical Study on the Impact of Positional Encoding in Transformer-based Monaural Speech Enhancementcited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 27; 2025: 23; 2026: 10.

60 papers.

Sort:

By area:

  1. 2024A Survey of Mambacited by 39
  2. 2024XLSR-Mamba: A Dual-Column Bidirectional State Space Model for Spoofing Attack Detectioncited by 30
  3. 2024Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesiscited by 23
  4. 2024Rethinking Mamba in Speech Processing by Self-Supervised Modelscited by 11
  5. 2025BiCrossMamba-ST: Speech Deepfake Detection with Bidirectional Mamba Spectro-Temporal Cross-Attentioncited by 10
  6. 2024Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Modelscited by 9
  7. 2024Mamba for Streaming ASR Combined with Unimodal Aggregationcited by 8
  8. 2024Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decodingcited by 7
  9. 2024SELD-Mamba: Selective State-Space Model for Sound Event Localization and Detection with Source Distance Estimationcited by 6
  10. 2024SlimSeiz: Efficient Channel-Adaptive Seizure Prediction Using a Mamba-Enhanced Networkcited by 6
  11. 2024SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Modelcited by 6
  12. 2025Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Losscited by 6
38 more
  1. 2025Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternativecited by 6
  2. 2024TF-Mamba: A Time-Frequency Network for Sound Source Localizationcited by 5
  3. 2024Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognitioncited by 5
  4. 2024Optimising TinyML with Quantization and Distillation of Transformer and Mamba Models for Indoor Localisation on Edge Devicescited by 5
  5. 2025The Alpha-Alternator: Dynamic Adaptation To Varying Noise Levels In Sequences Using The Vendi Score For Improved Robustness and Performancecited by 5
  6. 2024Vector Quantized Diffusion Model Based Speech Bandwidth Extensioncited by 4
  7. 2024Mamba-based Segmentation Model for Speaker Diarizationcited by 4
  8. 2024DocMamba: Efficient Document Pre-training with State Space Modelcited by 3
  9. 2024DeFT-Mamba: Universal Multichannel Sound Separation and Polyphonic Audio Classificationcited by 3
  10. 2025MemMamba: Rethinking Memory Patterns in State Space Modelcited by 3
  11. 2024An Analysis of Linear Complexity Attention Substitutes with BEST-RQcited by 2
  12. 2024EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignmentcited by 2
  13. 2024Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexitycited by 2
  14. 2024CleanUMamba: A Compact Mamba Network for Speech Denoising using Channel Pruningcited by 2
  15. 2024From KAN to GR-KAN: Advancing Speech Enhancement with KAN-Based Methodologycited by 2
  16. 2025Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractionscited by 2
  17. 2025Deep Active Speech Cancellation with Mamba-Masking Networkcited by 2
  18. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  19. 2025An Exploration of Mamba for Speech Self-Supervised Modelscited by 2
  20. 2026Improving Code-Switching Speech Recognition with TTS Data Augmentationcited by 2
  21. 2024ShadowMamba: State-Space Model with Boundary-Region Selective Scan for Shadow Removalcited by 1
  22. 2024Selective State Space Model for Monaural Speech Enhancementcited by 1
  23. 2024Long-context Protein Language Modeling Using Bidirectional Mamba with Shared Projection Layerscited by 1
  24. 2025Developing multilingual speech synthesis system for Ojibwe, Mi'kmaq, and Maliseetcited by 1
  25. 2025MLLM-based Speech Recognition: When and How is Multimodality Beneficial?cited by 1
  26. 2025MLMA: Towards Multilingual ASR With Mamba-based Architecturescited by 1
  27. 2025Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASRcited by 1
  28. 2026XLSR-MamBo: Scaling the Hybrid Mamba-Attention Backbone for Audio Deepfake Detectioncited by 1
  29. 2026MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesiscited by 1
  30. 2024DualKanbaFormer: An Efficient Selective Sparse Framework for Multimodal Aspect-based Sentiment Analysiscited by 0
  31. 2024aTENNuate: Optimized Real-time Speech Enhancement with Deep SSMs on Raw Audiocited by 0
  32. 2025Active Speech Enhancement: Active Speech Denoising Decliping and Deveraberationcited by 0
  33. 2025NTU Speechlab LLM-Based Multilingual ASR System for Interspeech MLC-SLM Challenge 2025cited by 0
  34. 2025Recognizing Dementia from Neuropsychological Tests with State Space Modelscited by 0
  35. 2025Keyword Mamba: Spoken Keyword Spotting with State Space Modelscited by 0
  36. 2025ESTM: An Enhanced Dual-Branch Spectral-Temporal Mamba for Anomalous Sound Detectioncited by 0
  37. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrogramscited by 0
  38. 2025Hazy Pedestrian Trajectory Prediction via Physical Priors and Graph-Mambacited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Mamba-SEUNet: Mamba UNet for Monaural Speech Enhancement
  2. 2024Exploring the Capability of Mamba in Speech Applications
  3. 2026Omni-directional attention mechanism based on Mamba for speech separation
  4. 2024Demystify Mamba in Vision: A Linear Attention Perspective
  5. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
  6. 2024MC-SEMamba: A Simple Multi-channel Extension of SEMamba
  7. 2024The Hidden Attention of Mamba Models
  8. 2024Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mamba