# Adaptive Discounting of Implicit Language Models in RNN-Transducers

arXiv: https://arxiv.org/abs/2203.02317 · HTML: https://mlsanity.com/paper/2203.02317/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 21 Feb 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [Cold Fusion: Training Seq2Seq Models Together with Language Models](https://mlsanity.com/paper/1708.06426/) (2017)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability](https://mlsanity.com/paper/2007.15188/) (2020)
- [Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2102.01380/) (2021)
- [Improving Tail Performance of a Deliberation E2E ASR Model Using a Large Text Corpus](https://mlsanity.com/paper/2008.10491/) (2020)
- [Fast Text-Only Domain Adaptation of RNN-Transducer Prediction Network](https://mlsanity.com/paper/2104.11127/) (2021)
- [Improving the fusion of acoustic and text representations in RNN-T](https://mlsanity.com/paper/2201.10240/) (2022)
- [Improving accuracy of rare words for RNN-Transducer through unigram shallow fusion](https://mlsanity.com/paper/2012.00133/) (2020)
- [Acoustic-To-Word Model Without OOV](https://mlsanity.com/paper/1711.10136/) (2017)

## Built on by (most cited first, 5 of 5)

- [Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition models](https://mlsanity.com/paper/2212.01992/) (2022)
- [Improving RNN-Transducers with Acoustic LookAhead](https://mlsanity.com/paper/2307.05006/) (2023)
- [Lost in Transcription: How Speech-to-Text Errors Derail Code Understanding](https://mlsanity.com/paper/2601.15339/) (2026)
- [CodeVaani: A Multilingual, Voice-Based Code Learning Assistant](https://mlsanity.com/paper/2511.20654/) (2025)
- [The Anatomy of an ASR Hallucination](https://mlsanity.com/paper/2609.04404/) (2026)

## Abstract (arXiv)

RNN-Transducer (RNN-T) models have become synonymous with streaming end-to-end ASR systems. While they perform competitively on a number of evaluation categories, rare words pose a serious challenge to RNN-T models. One main reason for the degradation in performance on rare words is that the language model (LM) internal to RNN-Ts can become overconfident and lead to hallucinated predictions that are acoustically inconsistent with the underlying speech. To address this issue, we propose a lightweight adaptive LM discounting technique AdaptLMD, that can be used with any RNN-T architecture without requiring any external resources or additional parameters. AdaptLMD uses a two-pronged approach: 1) Randomly mask the prediction network output to encourage the RNN-T to not be overly reliant on it's outputs. 2) Dynamically choose when to discount the implicit LM (ILM) based on rarity of recently predicted tokens and divergence between ILM and implicit acoustic model (IAM) scores. Comparing AdaptLMD to a competitive RNN-T baseline, we obtain up to 4% and 14% relative reductions in overall WER and rare word PER, respectively, on a conversational, code-mixed Hindi-English ASR task.