# Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Data

arXiv: https://arxiv.org/abs/2010.12096 · HTML: https://mlsanity.com/paper/2010.12096/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2020.

## Built on by (most cited first, 16 of 16)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Pseudo Label Is Better Than Human Label](https://mlsanity.com/paper/2203.12668/) (2022)
- [Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models](https://mlsanity.com/paper/2110.03334/) (2021)
- [Variable Attention Masking for Configurable Transformer Transducer Speech Recognition](https://mlsanity.com/paper/2211.01438/) (2022)
- [Improving the fusion of acoustic and text representations in RNN-T](https://mlsanity.com/paper/2201.10240/) (2022)
- [Input Length Matters: Improving RNN-T and MWER Training for Long-form Telephony Speech Recognition](https://mlsanity.com/paper/2110.03841/) (2021)
- [Improving Deliberation by Text-Only and Semi-Supervised Training](https://mlsanity.com/paper/2206.14716/) (2022)
- [Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation](https://mlsanity.com/paper/2211.00508/) (2022)
- [Improving Noisy Student Training on Non-target Domain Data for Automatic Speech Recognition](https://mlsanity.com/paper/2211.04717/) (2022)
- [Effective Context in Neural Speech Models](https://mlsanity.com/paper/2505.22487/) (2025)
- [Biased Self-supervised learning for ASR](https://mlsanity.com/paper/2211.02536/) (2022)
- [NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Training](https://mlsanity.com/paper/2409.08680/) (2024)
- [Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer](https://mlsanity.com/paper/2308.16415/) (2023)
- [REALM: Retrospective Encoder Alignment for LFP Modeling](https://mlsanity.com/paper/2605.14867/) (2026)
- [Alignment-Path Distillation from Non-streaming ASR-LLMs for Streaming Speech Recognition](https://mlsanity.com/paper/2609.20121/) (2026)

## Abstract (arXiv)

Streaming end-to-end automatic speech recognition (ASR) models are widely used on smart speakers and on-device applications. Since these models are expected to transcribe speech with minimal latency, they are constrained to be causal with no future context, compared to their non-streaming counterparts. Consequently, streaming models usually perform worse than non-streaming models. We propose a novel and effective learning method by leveraging a non-streaming ASR model as a teacher to generate transcripts on an arbitrarily large data set, which is then used to distill knowledge into streaming ASR models. This way, we scale the training of streaming models to up to 3 million hours of YouTube audio. Experiments show that our approach can significantly reduce the word error rate (WER) of RNNT models not only on LibriSpeech but also on YouTube data in four languages. For example, in French, we are able to reduce the WER by 16.4% relatively to a baseline streaming model by leveraging a non-streaming teacher model trained on the same amount of labeled data as the baseline.