# Improved Multi-Stage Training of Online Attention-based Encoder-Decoder Models

arXiv: https://arxiv.org/abs/1912.12384 · HTML: https://mlsanity.com/paper/1912.12384/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2019.

## Built on by (most cited first, 5 of 5)

- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021)
- [Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition](https://mlsanity.com/paper/2210.00169/) (2022)
- [Two-Pass End-to-End ASR Model Compression](https://mlsanity.com/paper/2201.02741/) (2022)
- [Data-driven grapheme-to-phoneme representations for a lexicon-free text-to-speech](https://mlsanity.com/paper/2401.10465/) (2024)
- [On the compression of shallow non-causal ASR models using knowledge distillation and tied-and-reduced decoder for low-latency on-device speech recognition](https://mlsanity.com/paper/2312.09842/) (2023)

## Abstract (arXiv)

In this paper, we propose a refined multi-stage multi-task training strategy to improve the performance of online attention-based encoder-decoder (AED) models. A three-stage training based on three levels of architectural granularity namely, character encoder, byte pair encoding (BPE) based encoder, and attention decoder, is proposed. Also, multi-task learning based on two-levels of linguistic granularity namely, character and BPE, is used. We explore different pre-training strategies for the encoders including transfer learning from a bidirectional encoder. Our encoder-decoder models with online attention show 35% and 10% relative improvement over their baselines for smaller and bigger models, respectively. Our models achieve a word error rate (WER) of 5.04% and 4.48% on the Librispeech test-clean data for the smaller and bigger models respectively after fusion with long short-term memory (LSTM) based external language model (LM).