# Librispeech Transducer Model with Internal Language Model Prior Correction arXiv: https://arxiv.org/abs/2104.03006 ยท HTML: https://mlsanity.com/paper/2104.03006/ ## Key facts - Cited by 26 papers in the map (data through 14 Oct 2026). - Builds on 17 of its 28 references that are in the map. - First posted on arXiv 7 Apr 2021. ## Builds on (most cited first, 17 of 17) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020) - [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013) - [Horovod: fast and easy distributed deep learning in TensorFlow](https://mlsanity.com/paper/1802.05799/) (2018) - [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020) - [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015) - [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020) - [ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context](https://mlsanity.com/paper/2005.03191/) (2020) - [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020) - [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020) - [RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentation](https://mlsanity.com/paper/1905.03072/) (2019) - [Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations](https://mlsanity.com/paper/1606.01305/) (2016) - [Improved training of end-to-end attention models for speech recognition](https://mlsanity.com/paper/1805.03294/) (2018) - [A New Training Pipeline for an Improved Neural Transducer](https://mlsanity.com/paper/2005.09319/) (2020) - [RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition](https://mlsanity.com/paper/1805.05225/) (2018) - [Phoneme Based Neural Transducer for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/2010.16368/) (2020) ## Built on by (most cited first, 26 of 26) - [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021) - [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023) - [Ming-Omni: A Unified Multimodal Model for Perception and Generation](https://mlsanity.com/paper/2506.09344/) (2025) - [Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation](https://mlsanity.com/paper/2510.24821/) (2025) - [M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance](https://mlsanity.com/paper/2502.18778/) (2025) - [Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.05354/) (2021) - [On Language Model Integration for RNN Transducer based Speech Recognition](https://mlsanity.com/paper/2110.06841/) (2021) - [Modular Hybrid Autoregressive Transducer](https://mlsanity.com/paper/2210.17049/) (2022) - [Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation](https://mlsanity.com/paper/2511.05516/) (2025) - [Internal Language Model Estimation Through Explicit Context Vector Learning for Attention-based Encoder-decoder ASR](https://mlsanity.com/paper/2201.11627/) (2022) - [JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognition](https://mlsanity.com/paper/2302.08583/) (2023) - [Comparing the Benefit of Synthetic Training Data for Various Automatic Speech Recognition Architectures](https://mlsanity.com/paper/2104.05379/) (2021) - [Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognition](https://mlsanity.com/paper/2309.08436/) (2023) - [Internal Language Model Estimation based Language Model Fusion for Cross-Domain Code-Switching Speech Recognition](https://mlsanity.com/paper/2207.04176/) (2022) - [PM-MMUT: Boosted Phone-Mask Data Augmentation using Multi-Modeling Unit Training for Phonetic-Reduction-Robust E2E Speech Recognition](https://mlsanity.com/paper/2112.06721/) (2021) - [Effective internal language model training and fusion for factorized transducer model](https://mlsanity.com/paper/2404.01716/) (2024) - [Channel-Aware Domain-Adaptive Generative Adversarial Network for Robust Speech Recognition](https://mlsanity.com/paper/2409.12386/) (2024) - [Back from the future: bidirectional CTC decoding using future information in speech recognition](https://mlsanity.com/paper/2110.03326/) (2021) - [Internal Language Model Estimation based Adaptive Language Model Fusion for Domain Adaptation](https://mlsanity.com/paper/2211.00968/) (2022) - [Improving Language Model Integration for Neural Machine Translation](https://mlsanity.com/paper/2306.05077/) (2023) - [Label-Context-Dependent Internal Language Model Estimation for CTC](https://mlsanity.com/paper/2506.06096/) (2025) - [AutoLV: Automatic Lecture Video Generator](https://mlsanity.com/paper/2209.08795/) (2022) - [Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptation](https://mlsanity.com/paper/2309.07369/) (2023) - [OrdMoE: Preference Alignment via Hierarchical Expert Group Ranking in Multimodal Mixture-of-Experts LLMs](https://mlsanity.com/paper/2511.19023/) (2025) - [Universal Robust Speech Adaptation for Cross-Domain Speech Recognition and Enhancement](https://mlsanity.com/paper/2602.04307/) (2026) - [Diffusion Language Models for Speech Recognition](https://mlsanity.com/paper/2604.14001/) (2026) ## Abstract (arXiv) We present our transducer model on Librispeech. We study variants to include an external language model (LM) with shallow fusion and subtract an estimated internal LM. This is justified by a Bayesian interpretation where the transducer model prior is given by the estimated internal LM. The subtraction of the internal LM gives us over 14% relative improvement over normal shallow fusion. Our transducer has a separate probability distribution for the non-blank labels which allows for easier combination with the external LM, and easier estimation of the internal LM. We additionally take care of including the end-of-sentence (EOS) probability of the external LM in the last blank probability which further improves the performance. All our code and setups are published.