# On Addressing Practical Challenges for RNN-Transducer

arXiv: https://arxiv.org/abs/2105.00858 · HTML: https://mlsanity.com/paper/2105.00858/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 27 Apr 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset](https://mlsanity.com/paper/2010.11395/) (2020)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015)
- [Transformer-Transducer: End-to-End Speech Recognition with Self-Attention](https://mlsanity.com/paper/1910.12977/) (2019)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition](https://mlsanity.com/paper/2005.14327/) (2020)
- [Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability](https://mlsanity.com/paper/2007.15188/) (2020)
- [Exploring Neural Transducers for End-to-End Speech Recognition](https://mlsanity.com/paper/1707.07413/) (2017)
- [A New Training Pipeline for an Improved Neural Transducer](https://mlsanity.com/paper/2005.09319/) (2020)
- [Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2001.01798/) (2020)
- [Learning Word-Level Confidence For Subword End-to-End ASR](https://mlsanity.com/paper/2103.06716/) (2021)
- [An evaluation of word-level confidence estimation for end-to-end automatic speech recognition](https://mlsanity.com/paper/2101.05525/) (2021)
- [Improving Performance of End-to-End ASR on Numeric Sequences](https://mlsanity.com/paper/1907.01372/) (2019)
- [Advancing Acoustic-to-Word CTC Model](https://mlsanity.com/paper/1803.05566/) (2018)
- [Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Text](https://mlsanity.com/paper/1905.01152/) (2019)
- [High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model](https://mlsanity.com/paper/2003.07482/) (2020)
- [Improving the Performance of Online Neural Transducer Models](https://mlsanity.com/paper/1712.01807/) (2017)
- [Modeling Multi-speaker Latent Space to Improve Neural TTS: Quick Enrolling New Speaker and Enhancing Premium Voice](https://mlsanity.com/paper/1812.05253/) (2018)

## Built on by (most cited first, 19 of 19)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition](https://mlsanity.com/paper/2301.07851/) (2023)
- [Anatomy of Industrial Scale Multilingual ASR](https://mlsanity.com/paper/2404.09841/) (2024)
- [Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition models](https://mlsanity.com/paper/2212.01992/) (2022)
- [Code-Switching Text Generation and Injection in Mandarin-English ASR](https://mlsanity.com/paper/2303.10949/) (2023)
- [How to Estimate Model Transferability of Pre-Trained Speech Models?](https://mlsanity.com/paper/2306.01015/) (2023)
- [Speech collage: code-switched audio generation by collaging monolingual corpora](https://mlsanity.com/paper/2309.15674/) (2023)
- [Word Level Timestamp Generation for Automatic Speech Recognition and Translation](https://mlsanity.com/paper/2505.15646/) (2025)
- [External Language Model Integration for Factorized Neural Transducers](https://mlsanity.com/paper/2305.17304/) (2023)
- [DistilXLSR: A Light Weight Cross-Lingual Speech Representation Model](https://mlsanity.com/paper/2306.01303/) (2023)
- [Improving Code-Switching and Named Entity Recognition in ASR with Speech Editing based Data Augmentation](https://mlsanity.com/paper/2306.08588/) (2023)
- [End-to-end speech recognition modeling from de-identified data](https://mlsanity.com/paper/2207.05469/) (2022)
- [Pronunciation-aware unique character encoding for RNN Transducer-based Mandarin speech recognition](https://mlsanity.com/paper/2207.14578/) (2022)
- [Partitioned Gradient Matching-based Data Subset Selection for Compute-Efficient Robust ASR Training](https://mlsanity.com/paper/2210.16892/) (2022)
- [Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptation](https://mlsanity.com/paper/2309.07369/) (2023)
- [Target word activity detector: An approach to obtain ASR word boundaries without lexicon](https://mlsanity.com/paper/2409.13913/) (2024)
- [CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translation](https://mlsanity.com/paper/2410.05146/) (2024)
- [Towards Personalization of CTC Speech Recognition Models with Contextual Adapters and Adaptive Boosting](https://mlsanity.com/paper/2210.09510/) (2022)
- [Incorporating Class-based Language Model for Named Entity Recognition in Factorized Neural Transducer](https://mlsanity.com/paper/2309.07648/) (2023)

## Abstract (arXiv)

In this paper, several works are proposed to address practical challenges for deploying RNN Transducer (RNN-T) based speech recognition system. These challenges are adapting a well-trained RNN-T model to a new domain without collecting the audio data, obtaining time stamps and confidence scores at word level. The first challenge is solved with a splicing data method which concatenates the speech segments extracted from the source domain data. To get the time stamp, a phone prediction branch is added to the RNN-T model by sharing the encoder for the purpose of force alignment. Finally, we obtain word-level confidence scores by utilizing several types of features calculated during decoding and from confusion network. Evaluated with Microsoft production data, the splicing data adaptation method improves the baseline and adaptation with the text to speech method by 58.03% and 15.25% relative word error rate reduction, respectively. The proposed time stamping method can get less than 50ms word timing difference from the ground truth alignment on average while maintaining the recognition accuracy of the RNN-T model. We also obtain high confidence annotation performance with limited computation cost.