# Improved training for online end-to-end speech recognition systems

arXiv: https://arxiv.org/abs/1711.02212 · HTML: https://mlsanity.com/paper/1711.02212/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2017.

## Built on by (most cited first, 13 of 13)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021)
- [Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latency](https://mlsanity.com/paper/2104.02176/) (2021)
- [Collaborative Training of Acoustic Encoders for Speech Recognition](https://mlsanity.com/paper/2106.08960/) (2021)
- [Anticipation-Free Training for Simultaneous Machine Translation](https://mlsanity.com/paper/2201.12868/) (2022)
- [Two-Pass End-to-End ASR Model Compression](https://mlsanity.com/paper/2201.02741/) (2022)
- [Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models](https://mlsanity.com/paper/2104.14346/) (2021)
- [A bandit approach to curriculum generation for automatic speech recognition](https://mlsanity.com/paper/2102.03662/) (2021)
- [Comparison and Analysis of New Curriculum Criteria for End-to-End ASR](https://mlsanity.com/paper/2208.05782/) (2022)
- [Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer](https://mlsanity.com/paper/2308.16415/) (2023)
- [Curriculum optimization for low-resource speech recognition](https://mlsanity.com/paper/2202.08883/) (2022)
- [Mutual Learning of Single- and Multi-Channel End-to-End Neural Diarization](https://mlsanity.com/paper/2210.03459/) (2022)

## Abstract (arXiv)

Achieving high accuracy with end-to-end speech recognizers requires careful parameter initialization prior to training. Otherwise, the networks may fail to find a good local optimum. This is particularly true for online networks, such as unidirectional LSTMs. Currently, the best strategy to train such systems is to bootstrap the training from a tied-triphone system. However, this is time consuming, and more importantly, is impossible for languages without a high-quality pronunciation lexicon. In this work, we propose an initialization strategy that uses teacher-student learning to transfer knowledge from a large, well-trained, offline end-to-end speech recognition model to an online end-to-end model, eliminating the need for a lexicon or any other linguistic resources. We also explore curriculum learning and label smoothing and show how they can be combined with the proposed teacher-student learning for further improvements. We evaluate our methods on a Microsoft Cortana personal assistant task and show that the proposed method results in a 19 % relative improvement in word error rate compared to a randomly-initialized baseline system.