mlsanityData to 14 Oct 2026

Mamba for Streaming ASR Combined with Unimodal Aggregation

arXiv 2410.00070 ↗First posted 30 Sep 2024eess.AS cs.CL cs.SDcited by 8

On the map: State-space sequence models (S4, Mamba) → Speech and audio SSMs

Abstract (arXiv)

This paper works on streaming automatic speech recognition (ASR). Mamba, a recently proposed state space model, has demonstrated the ability to match or surpass Transformers in various tasks while benefiting from a linear complexity advantage. We explore the efficiency of Mamba encoder for streaming ASR and propose an associated lookahead mechanism for leveraging controllable future information. Additionally, a streaming-style unimodal aggregation (UMA) method is implemented, which automatically detects token activity and streamingly triggers token output, and meanwhile aggregates feature frames for better learning token representation. Based on UMA, an early termination (ET) method is proposed to further reduce recognition latency. Experiments conducted on two Mandarin Chinese datasets demonstrate that the proposed model achieves competitive ASR performance in terms of both recognition accuracy and latency.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 21 of its 26 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

21 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis13 Jul
  2. 2024Mamba in Speech: Towards an Alternative to Self-Attention21 May

Other references · 19

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  4. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  5. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  6. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  7. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 128
  8. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 77
  9. 2020Streaming automatic speech recognition with the transformer modelcited by 73
  10. 2020FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularizationcited by 69
  11. 2020Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognitioncited by 39
  12. 2019Transformer ASR with Contextual Block Processingcited by 28
7 more
  1. 2022Delay-penalized transducer for low-latency streaming ASRcited by 7
  2. 2022TrimTail: Low-Latency Streaming ASR with Simple but Effective Spectrogram-Level Length Penaltycited by 6
  3. 2022Peak-First CTC: Reducing the Peak Latency of CTC Models by Applying Peak-First Regularizationcited by 6
  4. 2023Lookahead When It Matters: Adaptive Non-causal Transformers for Streaming Neural Transducerscited by 6
  5. 2024Streaming Decoder-Only Automatic Speech Recognition with Discrete Speech Units: A Pilot Studycited by 6
  6. 2022Transformer-based Streaming ASR with Cumulative Attentioncited by 3
  7. 2023Unimodal Aggregation for CTC-based Speech Recognitioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 1.

8 papers.

  1. 2024High-precision medical speech recognition through synthetic data and semantic correction: UNITED-MEDASRcited by 2
  2. 2025An Exploration of Mamba for Speech Self-Supervised Modelscited by 2
  3. 2025Delayed-KD: Delayed Knowledge Distillation based CTC for Low-Latency Streaming ASRcited by 1
  4. 2025Towards Streaming LiDAR Object Detection with Point Clouds as Egocentric Sequencescited by 1
  5. 2025MLMA: Towards Multilingual ASR With Mamba-based Architecturescited by 1
  6. 2025UMA-Split: unimodal aggregation for both English and Mandarin non-autoregressive speech recognitioncited by 0
  7. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrogramscited by 0
  8. 2026MambAdapter: Lightweight Mamba-Based Adapters for Parameter-Efficient Transfer Learning in Speech and Audiocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Joint Optimization of Streaming and Non-Streaming Automatic Speech Recognition with Multi-Decoder and Knowledge Distillation
  2. 2023Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer
  3. 2026From Monolingual to Multilingual: Evaluating Mamba for ASR in South African Languages
  4. 2026TC-BiMamba: Trans-Chunk bidirectionally within BiMamba for unified streaming and non-streaming ASR
  5. 2021Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models
  6. 2022Improving Streaming End-to-End ASR on Transformer-based Causal Models with Encoder States Revision Strategies
  7. 2024Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognition
  8. 2025Samba-ASR: State-Of-The-Art Speech Recognition Leveraging Structured State-Space Models