mlsanityData to 14 Oct 2026

Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognition

arXiv 2411.06968 ↗First posted 11 Nov 2024cs.SD eess.AScited by 5

On the map: State-space sequence models (S4, Mamba) → Speech and audio SSMs

Abstract (arXiv)

Selective state space models (SSMs) represented by Mamba have demonstrated their computational efficiency and promising outcomes in various tasks, including automatic speech recognition (ASR). Mamba has been applied to ASR task with the attention-based encoder-decoder framework, where the cross-attention mechanism between encoder and decoder remains. This paper explores the capability of Mamba as the decoder-only architecture in ASR task. Our MAmba-based DEcoder-ONly approach (MADEON) consists of a single decoder that takes speech tokens as a condition and predicts text tokens in an autoregressive manner. To enhance MADEON, we further propose speech prefixing that performs bidirectional processing on speech tokens, which enriches the contextual information in the hidden states. Our experiments show that MADEON significantly outperforms a non-selective SSM. The combination of speech prefixing and the recently proposed Mamba-2 yields comparable performance to Transformer-based models on large datasets.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 43 of its 49 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

43 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Exploring the Capability of Mamba in Speech Applications24 Jun
  2. 2024Mamba in Speech: Towards an Alternative to Self-Attention21 May
  3. 2024An Investigation of Incorporating Mamba for Speech Enhancement10 May
  4. 2023Augmenting conformers with structured state-space sequence models for online speech recognition15 Sep
  5. 2023A Multi-dimensional Deep Structured State Space Approach to Speech Enhancement Using Small-footprint Models1 Jun
  6. 2022Structured State Space Decoder for Speech Recognition and Synthesis31 Oct

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  9. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  10. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  11. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
25 more
  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  2. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  3. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  4. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  5. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  6. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  7. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  8. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  9. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  10. 2021Direct speech-to-speech translation with discrete unitscited by 149
  11. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  12. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  13. 2024Mamba-ND: Selective State Space Modeling for Multi-Dimensional Datacited by 84
  14. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  15. 2024MambaByte: Token-free Selective State Space Modelcited by 59
  16. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  17. 2022WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Modelscited by 26
  18. 2023Diagonal State Space Augmented Transformers for Speech Recognitioncited by 25
  19. 2023Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTScited by 24
  20. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  21. 2023A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Taskscited by 14
  22. 2023Multi-Head State Space Model for Speech Recognitioncited by 14
  23. 2023Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentationcited by 10
  24. 2023Loss Masking Is Not Needed in Decoder-only Transformer for Discrete-token-based ASRcited by 4
  25. 2023Multiple Representation Transfer from Large Language Models to End-to-End ASR Systemscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 5.

5 papers.

  1. 2025Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASRcited by 0
  2. 2025Keyword Mamba: Spoken Keyword Spotting with State Space Modelscited by 0
  3. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrogramscited by 0
  4. 2025ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extractioncited by 0
  5. 2025PSMamba: Progressive Self-supervised Vision Mamba for Plant Disease Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models
  2. 2026Omni-directional attention mechanism based on Mamba for speech separation
  3. 2025An Exploration of Mamba for Speech Self-Supervised Models
  4. 2024Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis
  5. 2024Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation
  6. 2024Rethinking Mamba in Speech Processing by Self-Supervised Models
  7. 2024BEST-STD: Bidirectional Mamba-Enhanced Speech Tokenization for Spoken Term Detection
  8. 2026From Monolingual to Multilingual: Evaluating Mamba for ASR in South African Languages