# Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognition

arXiv: https://arxiv.org/abs/2007.13802 · HTML: https://mlsanity.com/paper/2007.13802/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2020.

## Built on by (most cited first, 30 of 34)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Tied & Reduced RNN-T Decoder](https://mlsanity.com/paper/2109.07513/) (2021)
- [Personalization Strategies for End-to-End Speech Recognition Systems](https://mlsanity.com/paper/2102.07739/) (2021)
- [Wav2vec-C: A Self-supervised Model for Speech Representation Learning](https://mlsanity.com/paper/2103.08393/) (2021)
- [JOIST: A Joint Speech and Text Streaming Model For ASR](https://mlsanity.com/paper/2210.07353/) (2022)
- [Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition](https://mlsanity.com/paper/2106.02302/) (2021)
- [Efficient Training of Neural Transducer for Speech Recognition](https://mlsanity.com/paper/2204.10586/) (2022)
- [Improving Rare Word Recognition with LM-aware MWER Training](https://mlsanity.com/paper/2204.07553/) (2022)
- [Minimising Biasing Word Errors for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2205.09058/) (2022)
- [Input Length Matters: Improving RNN-T and MWER Training for Long-form Telephony Speech Recognition](https://mlsanity.com/paper/2110.03841/) (2021)
- [Do as I mean, not as I say: Sequence Loss Training for Spoken Language Understanding](https://mlsanity.com/paper/2102.06750/) (2021)
- [Residual Energy-Based Models for End-to-End Speech Recognition](https://mlsanity.com/paper/2103.14152/) (2021)
- [Consistent Training and Decoding For End-to-end Speech Recognition Using Lattice-free MMI](https://mlsanity.com/paper/2112.02498/) (2021)
- [Self-critical Sequence Training for Automatic Speech Recognition](https://mlsanity.com/paper/2204.06260/) (2022)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [Powerful and Extensible WFST Framework for RNN-Transducer Losses](https://mlsanity.com/paper/2303.10384/) (2023)
- [On joint training with interfaces for spoken language understanding](https://mlsanity.com/paper/2106.15919/) (2021)
- [A Likelihood Ratio based Domain Adaptation Method for E2E Models](https://mlsanity.com/paper/2201.03655/) (2022)
- [Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition](https://mlsanity.com/paper/2412.16464/) (2024)
- [Flexi-Transducer: Optimizing Latency, Accuracy and Compute forMulti-Domain On-Device Scenarios](https://mlsanity.com/paper/2104.02232/) (2021)
- [Integrating Lattice-Free MMI into End-to-End Speech Recognition](https://mlsanity.com/paper/2203.15614/) (2022)
- [Memory-Efficient Training of RNN-Transducer with Sampled Softmax](https://mlsanity.com/paper/2203.16868/) (2022)
- [Better Transcription of UK Supreme Court Hearings](https://mlsanity.com/paper/2211.17094/) (2022)
- [Lattice-Free Sequence Discriminative Training for Phoneme-Based Neural Transducers](https://mlsanity.com/paper/2212.04325/) (2022)
- [Effective internal language model training and fusion for factorized transducer model](https://mlsanity.com/paper/2404.01716/) (2024)
- [Towards scalable efficient on-device ASR with transfer learning](https://mlsanity.com/paper/2407.16664/) (2024)
- [Multiple-hypothesis RNN-T Loss for Unsupervised Fine-tuning and Self-training of Neural Transducer](https://mlsanity.com/paper/2207.14736/) (2022)
- [On the Relation between Internal Language Model and Sequence Discriminative Training for Neural Transducers](https://mlsanity.com/paper/2309.14130/) (2023)
- [Automatic Learning of Subword Dependent Model Scales](https://mlsanity.com/paper/2110.09324/) (2021)
- [Record Deduplication for Entity Distribution Modeling in ASR Transcripts](https://mlsanity.com/paper/2306.06246/) (2023)

## Abstract (arXiv)

In this work, we propose a novel and efficient minimum word error rate (MWER) training method for RNN-Transducer (RNN-T). Unlike previous work on this topic, which performs on-the-fly limited-size beam-search decoding and generates alignment scores for expected edit-distance computation, in our proposed method, we re-calculate and sum scores of all the possible alignments for each hypothesis in N-best lists. The hypothesis probability scores and back-propagated gradients are calculated efficiently using the forward-backward algorithm. Moreover, the proposed method allows us to decouple the decoding and training processes, and thus we can perform offline parallel-decoding and MWER training for each subset iteratively. Experimental results show that this proposed semi-on-the-fly method can speed up the on-the-fly method by 6 times and result in a similar WER improvement (3.6%) over a baseline RNN-T model. The proposed MWER training can also effectively reduce high-deletion errors (9.2% WER-reduction) introduced by RNN-T models when EOS is added for endpointer. Further improvement can be achieved if we use a proposed RNN-T rescoring method to re-rank hypotheses and use external RNN-LM to perform additional rescoring. The best system achieves a 5% relative improvement on an English test-set of real far-field recordings and a 11.6% WER reduction on music-domain utterances.