# RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions

arXiv: https://arxiv.org/abs/2005.03271 · HTML: https://mlsanity.com/paper/2005.03271/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2020.

## Built on by (most cited first, 26 of 26)

- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Pseudo-Labeling for Massively Multilingual Speech Recognition](https://mlsanity.com/paper/2111.00161/) (2021)
- [E2E Segmenter: Joint Segmenting and Decoding for Long-Form ASR](https://mlsanity.com/paper/2204.10749/) (2022)
- [Scaling ASR Improves Zero and Few Shot Learning](https://mlsanity.com/paper/2111.05948/) (2021)
- [Efficient Domain Adaptation for Speech Foundation Models](https://mlsanity.com/paper/2302.01496/) (2023)
- [Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study](https://mlsanity.com/paper/2401.12789/) (2024)
- [Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planning](https://mlsanity.com/paper/2506.17525/) (2025)
- [Input Length Matters: Improving RNN-T and MWER Training for Long-form Telephony Speech Recognition](https://mlsanity.com/paper/2110.03841/) (2021)
- [Multi-Modal Pre-Training for Automated Speech Recognition](https://mlsanity.com/paper/2110.09890/) (2021)
- [Updated Corpora and Benchmarks for Long-Form Speech Recognition](https://mlsanity.com/paper/2309.15013/) (2023)
- [Hypothesis Stitcher for End-to-End Speaker-attributed ASR on Long-form Multi-talker Recordings](https://mlsanity.com/paper/2101.01853/) (2021)
- [Investigating data partitioning strategies for crosslinguistic low-resource ASR evaluation](https://mlsanity.com/paper/2208.12888/) (2022)
- [Distribution Aware Metrics for Conditional Natural Language Generation](https://mlsanity.com/paper/2209.07518/) (2022)
- [Improving RNN-Transducers with Acoustic LookAhead](https://mlsanity.com/paper/2307.05006/) (2023)
- [Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models](https://mlsanity.com/paper/2104.14346/) (2021)
- [Smart Speech Segmentation using Acousto-Linguistic Features with look-ahead](https://mlsanity.com/paper/2210.14446/) (2022)
- [Random Utterance Concatenation Based Data Augmentation for Improving Short-video Speech Recognition](https://mlsanity.com/paper/2210.15876/) (2022)
- [E2E Segmentation in a Two-Pass Cascaded Encoder ASR Model](https://mlsanity.com/paper/2211.15432/) (2022)
- [Long-form Simultaneous Speech Translation: Thesis Proposal](https://mlsanity.com/paper/2310.11141/) (2023)
- [Generalizing RNN-Transducer to Out-Domain Audio via Sparse Self-Attention Layers](https://mlsanity.com/paper/2108.10752/) (2021)
- [DSS: Synthesizing long Digital Ink using Data augmentation, Style encoding and Split generation](https://mlsanity.com/paper/2311.17786/) (2023)
- [Improved Long-Form Speech Recognition by Jointly Modeling the Primary and Non-primary Speakers](https://mlsanity.com/paper/2312.11123/) (2023)
- [MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancement](https://mlsanity.com/paper/2507.00966/) (2025)
- [Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition](https://mlsanity.com/paper/2403.19822/) (2024)
- [Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers](https://mlsanity.com/paper/2502.05232/) (2025)

## Abstract (arXiv)

In recent years, all-neural end-to-end approaches have obtained state-of-the-art results on several challenging automatic speech recognition (ASR) tasks. However, most existing works focus on building ASR models where train and test data are drawn from the same domain. This results in poor generalization characteristics on mismatched-domains: e.g., end-to-end models trained on short segments perform poorly when evaluated on longer utterances. In this work, we analyze the generalization properties of streaming and non-streaming recurrent neural network transducer (RNN-T) based end-to-end models in order to identify model components that negatively affect generalization performance. We propose two solutions: combining multiple regularization techniques during training, and using dynamic overlapping inference. On a long-form YouTube test set, when the nonstreaming RNN-T model is trained with shorter segments of data, the proposed combination improves word error rate (WER) from 22.3% to 14.8%; when the streaming RNN-T model trained on short Search queries, the proposed techniques improve WER on the YouTube set from 67.0% to 25.3%. Finally, when trained on Librispeech, we find that dynamic overlapping inference improves WER on YouTube from 99.8% to 33.0%.