# Mamba for Streaming ASR Combined with Unimodal Aggregation

arXiv: https://arxiv.org/abs/2410.00070 · HTML: https://mlsanity.com/paper/2410.00070/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Speech and audio SSMs](https://mlsanity.com/ssm/speech-and-audio-ssms/)

## Builds on (most cited first, 21 of 21)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale](https://mlsanity.com/paper/1808.10583/) (2018)
- [CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition](https://mlsanity.com/paper/1905.11235/) (2019)
- [Streaming automatic speech recognition with the transformer model](https://mlsanity.com/paper/2001.02674/) (2020)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [Mamba in Speech: Towards an Alternative to Self-Attention](https://mlsanity.com/paper/2405.12609/) (2024)
- [Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition](https://mlsanity.com/paper/2012.05481/) (2020)
- [Transformer ASR with Contextual Block Processing](https://mlsanity.com/paper/1910.07204/) (2019)
- [Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis](https://mlsanity.com/paper/2407.09732/) (2024)
- [Delay-penalized transducer for low-latency streaming ASR](https://mlsanity.com/paper/2211.00490/) (2022)
- [TrimTail: Low-Latency Streaming ASR with Simple but Effective Spectrogram-Level Length Penalty](https://mlsanity.com/paper/2211.00522/) (2022)
- [Peak-First CTC: Reducing the Peak Latency of CTC Models by Applying Peak-First Regularization](https://mlsanity.com/paper/2211.03284/) (2022)
- [Lookahead When It Matters: Adaptive Non-causal Transformers for Streaming Neural Transducers](https://mlsanity.com/paper/2305.04159/) (2023)
- [Streaming Decoder-Only Automatic Speech Recognition with Discrete Speech Units: A Pilot Study](https://mlsanity.com/paper/2406.18862/) (2024)
- [Transformer-based Streaming ASR with Cumulative Attention](https://mlsanity.com/paper/2203.05736/) (2022)
- [Unimodal Aggregation for CTC-based Speech Recognition](https://mlsanity.com/paper/2309.08150/) (2023)

## Built on by (most cited first, 8 of 8)

- [High-precision medical speech recognition through synthetic data and semantic correction: UNITED-MEDASR](https://mlsanity.com/paper/2412.00055/) (2024)
- [An Exploration of Mamba for Speech Self-Supervised Models](https://mlsanity.com/paper/2506.12606/) (2025)
- [Delayed-KD: Delayed Knowledge Distillation based CTC for Low-Latency Streaming ASR](https://mlsanity.com/paper/2505.22069/) (2025)
- [Towards Streaming LiDAR Object Detection with Point Clouds as Egocentric Sequences](https://mlsanity.com/paper/2506.06944/) (2025)
- [MLMA: Towards Multilingual ASR With Mamba-based Architectures](https://mlsanity.com/paper/2510.18684/) (2025)
- [UMA-Split: unimodal aggregation for both English and Mandarin non-autoregressive speech recognition](https://mlsanity.com/paper/2509.14653/) (2025)
- [An overview of neural architectures for self-supervised audio representation learning from masked spectrograms](https://mlsanity.com/paper/2509.18691/) (2025)
- [MambAdapter: Lightweight Mamba-Based Adapters for Parameter-Efficient Transfer Learning in Speech and Audio](https://mlsanity.com/paper/2606.15638/) (2026)

## Abstract (arXiv)

This paper works on streaming automatic speech recognition (ASR). Mamba, a recently proposed state space model, has demonstrated the ability to match or surpass Transformers in various tasks while benefiting from a linear complexity advantage. We explore the efficiency of Mamba encoder for streaming ASR and propose an associated lookahead mechanism for leveraging controllable future information. Additionally, a streaming-style unimodal aggregation (UMA) method is implemented, which automatically detects token activity and streamingly triggers token output, and meanwhile aggregates feature frames for better learning token representation. Based on UMA, an early termination (ET) method is proposed to further reduce recognition latency. Experiments conducted on two Mandarin Chinese datasets demonstrate that the proposed model achieves competitive ASR performance in terms of both recognition accuracy and latency.