# Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducer

arXiv: https://arxiv.org/abs/2010.13878 · HTML: https://mlsanity.com/paper/2010.13878/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2020.

## Built on by (most cited first, 22 of 22)

- [Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion](https://mlsanity.com/paper/2104.02194/) (2021)
- [Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding](https://mlsanity.com/paper/2104.02138/) (2021)
- [Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2102.01380/) (2021)
- [Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2109.00627/) (2021)
- [Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network](https://mlsanity.com/paper/2305.12493/) (2023)
- [Dissecting User-Perceived Latency of On-Device E2E Speech Recognition](https://mlsanity.com/paper/2104.02207/) (2021)
- [Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognition](https://mlsanity.com/paper/2306.00804/) (2023)
- [Minimising Biasing Word Errors for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2205.09058/) (2022)
- [Optimizing Bilingual Neural Transducer with Synthetic Code-switching Text Generation](https://mlsanity.com/paper/2210.12214/) (2022)
- [Joint Audio/Text Training for Transformer Rescorer of Streaming Speech Recognition](https://mlsanity.com/paper/2211.00174/) (2022)
- [Private Language Model Adaptation for Speech Recognition](https://mlsanity.com/paper/2110.10026/) (2021)
- [Modeling Spoken Information Queries for Virtual Assistants: Open Problems, Challenges and Opportunities](https://mlsanity.com/paper/2304.13149/) (2023)
- [Bayesian Neural Network Language Modeling for Speech Recognition](https://mlsanity.com/paper/2208.13259/) (2022)
- [Contextual Density Ratio for Language Model Biasing of Sequence to Sequence ASR Systems](https://mlsanity.com/paper/2206.14623/) (2022)
- [Adaptive Multi-Corpora Language Model Training for Speech Recognition](https://mlsanity.com/paper/2211.05121/) (2022)
- [Modality Confidence Aware Training for Robust End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2307.12134/) (2023)
- [Augmenting text for spoken language understanding with Large Language Models](https://mlsanity.com/paper/2309.09390/) (2023)
- [Correction Focused Language Model Training for Speech Recognition](https://mlsanity.com/paper/2310.11003/) (2023)
- [Pronunciation-aware unique character encoding for RNN Transducer-based Mandarin speech recognition](https://mlsanity.com/paper/2207.14578/) (2022)
- [Contextualized Automatic Speech Recognition with Dynamic Vocabulary Prediction and Activation](https://mlsanity.com/paper/2505.23077/) (2025)
- [Automatic Text Pronunciation Correlation Generation and Application for Contextual Biasing](https://mlsanity.com/paper/2501.00804/) (2025)
- [SENS-ASR: Semantic Embedding injection in Neural-transducer for Streaming Automatic Speech Recognition](https://mlsanity.com/paper/2603.10005/) (2026)

## Abstract (arXiv)

Recurrent Neural Network Transducer (RNN-T), like most end-to-end speech recognition model architectures, has an implicit neural network language model (NNLM) and cannot easily leverage unpaired text data during training. Previous work has proposed various fusion methods to incorporate external NNLMs into end-to-end ASR to address this weakness. In this paper, we propose extensions to these techniques that allow RNN-T to exploit external NNLMs during both training and inference time, resulting in 13-18% relative Word Error Rate improvement on Librispeech compared to strong baselines. Furthermore, our methods do not incur extra algorithmic latency and allow for flexible plug-and-play of different NNLMs without re-training. We also share in-depth analysis to better understand the benefits of the different NNLM fusion methods. Our work provides a reliable technique for leveraging unpaired text data to significantly improve RNN-T while keeping the system streamable, flexible, and lightweight.