# Low Latency End-to-End Streaming Speech Recognition with a Scout Network

arXiv: https://arxiv.org/abs/2003.10369 · HTML: https://mlsanity.com/paper/2003.10369/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2020.

## Built on by (most cited first, 20 of 20)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR](https://mlsanity.com/paper/2106.06636/) (2021)
- [Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition](https://mlsanity.com/paper/2312.17279/) (2023)
- [Dissecting User-Perceived Latency of On-Device E2E Speech Recognition](https://mlsanity.com/paper/2104.02207/) (2021)
- [Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2107.01269/) (2021)
- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021)
- [Personal VAD 2.0: Optimizing Personal Voice Activity Detection for On-Device Speech Recognition](https://mlsanity.com/paper/2204.03793/) (2022)
- [Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition](https://mlsanity.com/paper/2309.07988/) (2023)
- [Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognition](https://mlsanity.com/paper/2309.08436/) (2023)
- [Streaming Transformer Transducer Based Speech Recognition Using Non-Causal Convolution](https://mlsanity.com/paper/2110.05241/) (2021)
- [Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation](https://mlsanity.com/paper/2103.09903/) (2021)
- [Flexi-Transducer: Optimizing Latency, Accuracy and Compute forMulti-Domain On-Device Scenarios](https://mlsanity.com/paper/2104.02232/) (2021)
- [Visualization: the missing factor in Simultaneous Speech Translation](https://mlsanity.com/paper/2111.00514/) (2021)
- [HMM-Free Encoder Pre-Training for Streaming RNN Transducer](https://mlsanity.com/paper/2104.10764/) (2021)
- [VAD-free Streaming Hybrid CTC/Attention ASR for Unsegmented Recording](https://mlsanity.com/paper/2107.07509/) (2021)
- [Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals](https://mlsanity.com/paper/2308.08125/) (2023)
- [ExKaldi-RT: A Real-Time Automatic Speech Recognition Extension Toolkit of Kaldi](https://mlsanity.com/paper/2104.01384/) (2021)
- [Streaming Speech-to-Confusion Network Speech Recognition](https://mlsanity.com/paper/2306.03778/) (2023)
- [WhisperRT -- Turning Whisper into a Causal Streaming Model](https://mlsanity.com/paper/2508.12301/) (2025)
- [StreamAlign: Streaming Text-Aligned Speech Tokenization](https://mlsanity.com/paper/2609.09719/) (2026)

## Abstract (arXiv)

The attention-based Transformer model has achieved promising results for speech recognition (SR) in the offline mode. However, in the streaming mode, the Transformer model usually incurs significant latency to maintain its recognition accuracy when applying a fixed-length look-ahead window in each encoder layer. In this paper, we propose a novel low-latency streaming approach for Transformer models, which consists of a scout network and a recognition network. The scout network detects the whole word boundary without seeing any future frames, while the recognition network predicts the next subword by utilizing the information from all the frames before the predicted boundary. Our model achieves the best performance (2.7/6.4 WER) with only 639 ms latency on the test-clean and test-other data sets of Librispeech.