# Integrating Text Inputs For Training and Adapting RNN Transducer ASR Models

arXiv: https://arxiv.org/abs/2202.13155 · HTML: https://mlsanity.com/paper/2202.13155/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 26 Feb 2022.

## Builds on (most cited first, 12 of 12)

- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [Joint Speech Recognition and Speaker Diarization via Sequence Transduction](https://mlsanity.com/paper/1907.05337/) (2019)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Fast Text-Only Domain Adaptation of RNN-Transducer Prediction Network](https://mlsanity.com/paper/2104.11127/) (2021)
- [HarperValleyBank: A Domain-Specific Spoken Dialog Corpus](https://mlsanity.com/paper/2010.13929/) (2020)
- [Integrating Dialog History into End-to-End Spoken Language Understanding Systems](https://mlsanity.com/paper/2108.08405/) (2021)

## Built on by (most cited first, 20 of 20)

- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation](https://mlsanity.com/paper/2310.09424/) (2023)
- [Retrieval Augmented End-to-End Spoken Dialog Models](https://mlsanity.com/paper/2402.01828/) (2024)
- [Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator](https://mlsanity.com/paper/2302.14036/) (2023)
- [Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding](https://mlsanity.com/paper/2306.07944/) (2023)
- [Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning](https://mlsanity.com/paper/2506.05671/) (2025)
- [Understanding Shared Speech-Text Representations](https://mlsanity.com/paper/2304.14514/) (2023)
- [Improving Large-scale Deep Biasing with Phoneme Features and Text-only Data in Streaming Transducer](https://mlsanity.com/paper/2311.08966/) (2023)
- [Text-only Domain Adaptation using Unified Speech-Text Representation in Transducer](https://mlsanity.com/paper/2306.04076/) (2023)
- [On-the-fly Text Retrieval for End-to-End ASR Adaptation](https://mlsanity.com/paper/2303.10942/) (2023)
- [Using Text Injection to Improve Recognition of Personal Identifiers in Speech](https://mlsanity.com/paper/2308.07393/) (2023)
- [Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing](https://mlsanity.com/paper/2309.15826/) (2023)
- [CJST: CTC Compressor based Joint Speech and Text Training for Decoder-Only ASR](https://mlsanity.com/paper/2411.07607/) (2024)
- [Advances in Set Function Learning: A Survey of Techniques and Applications](https://mlsanity.com/paper/2501.14991/) (2025)
- [Text Injection for Capitalization and Turn-Taking Prediction in Speech Models](https://mlsanity.com/paper/2308.07395/) (2023)
- [A Non-autoregressive Model for Joint STT and TTS](https://mlsanity.com/paper/2501.09104/) (2025)
- [Enhanced Hybrid Transducer and Attention Encoder Decoder with Text Data](https://mlsanity.com/paper/2506.19159/) (2025)
- [End-to-End Open Vocabulary Keyword Search With Multilingual Neural Representations](https://mlsanity.com/paper/2308.08027/) (2023)
- [Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis](https://mlsanity.com/paper/2310.05374/) (2023)
- [Written Term Detection Improves Spoken Term Detection](https://mlsanity.com/paper/2407.04601/) (2024)

## Abstract (arXiv)

Compared to hybrid automatic speech recognition (ASR) systems that use a modular architecture in which each component can be independently adapted to a new domain, recent end-to-end (E2E) ASR system are harder to customize due to their all-neural monolithic construction. In this paper, we propose a novel text representation and training framework for E2E ASR models. With this approach, we show that a trained RNN Transducer (RNN-T) model's internal LM component can be effectively adapted with text-only data. An RNN-T model trained using both speech and text inputs improves over a baseline model trained on just speech with close to 13% word error rate (WER) reduction on the Switchboard and CallHome test sets of the NIST Hub5 2000 evaluation. The usefulness of the proposed approach is further demonstrated by customizing this general purpose RNN-T model to three separate datasets. We observe 20-45% relative word error rate (WER) reduction in these settings with this novel LM style customization technique using only unpaired text data from the new domains.