mlsanityData to 14 Oct 2026

Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition

arXiv 2102.01380 ↗First posted 2 Feb 2021eess.AS cs.AI cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

The efficacy of external language model (LM) integration with existing end-to-end (E2E) automatic speech recognition (ASR) systems can be improved significantly using the internal language model estimation (ILME) method. In this method, the internal LM score is subtracted from the score obtained by interpolating the E2E score with the external LM score, during inference. To improve the ILME-based inference, we propose an internal LM training (ILMT) method to minimize an additional internal LM loss by updating only the E2E model components that affect the internal LM estimation. ILMT encourages the E2E model to form a standalone LM inside its existing components, without sacrificing ASR accuracy. After ILMT, the more modular E2E model with matched training and inference criteria enables a more thorough elimination of the source-domain internal LM, and therefore leads to a more effective integration of the target-domain external LM. Experimented with 30K-hour trained recurrent neural network transducer and attention-based encoder-decoder models, ILMT with ILME-based inference achieves up to 31.5% and 11.4% relative word error rate reductions from standard E2E training with Shallow Fusion on out-of-domain LibriSpeech and in-domain Microsoft production test sets, respectively.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 2 Feb 2021.

Builds on

29 of 46 references.

References · 29

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2015Listen, Attend and Spellcited by 543
  3. 2015Attention-Based Models for Speech Recognitioncited by 401
  4. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  5. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 211
  6. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  7. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 150
  8. 2020Hybrid Autoregressive Transducer (hat)cited by 108
  9. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  10. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 100
  11. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
  12. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
17 more
  1. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 63
  2. 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 63
  3. 2019Conditional Teacher-Student Learningcited by 34
  4. 2016Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognitioncited by 33
  5. 2018Adversarial Teacher-Student Learning for Unsupervised Domain Adaptationcited by 28
  6. 2020Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognitioncited by 25
  7. 2018Speaker-Invariant Training via Adversarial Learningcited by 23
  8. 2020Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducercited by 22
  9. 2019RNN-T For Latency Controlled ASR With Improved Beam Searchcited by 20
  10. 2018Advancing Acoustic-to-Word CTC Modelcited by 15
  11. 2016Invariant Representations for Noisy Speech Recognitioncited by 12
  12. 2019Speaker Adaptation for Attention-Based End-to-End Speech Recognitioncited by 12
  13. 2017Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognitioncited by 10
  14. 2017Deep Long Short-Term Memory Adaptive Beamforming Networks For Multichannel Robust Speech Recognitioncited by 10
  15. 2019Adversarial Speaker Adaptationcited by 10
  16. 2020L-Vector: Neural Label Embedding for Domain Adaptationcited by 9
  17. 2020Character-Aware Attention-Based End-to-End Speech Recognitioncited by 3

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 14; 2023: 11; 2024: 5; 2025: 1.

37 papers.

Sort:

By area:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  2. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  3. 2021Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognitioncited by 32
  4. 2021Factorized Neural Transducer for Efficient Language Model Adaptationcited by 32
  5. 2021Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognitioncited by 19
  6. 2021Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognitioncited by 17
  7. 2021On Language Model Integration for RNN Transducer based Speech Recognitioncited by 17
  8. 2022Modular Hybrid Autoregressive Transducercited by 17
  9. 2022Towards Zero-Shot Code-Switched Speech Recognitioncited by 15
  10. 2022Improving the fusion of acoustic and text representations in RNN-Tcited by 11
  11. 2022Minimising Biasing Word Errors for Contextual ASR with the Tree-Constrained Pointer Generatorcited by 11
  12. 2022Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition modelscited by 11
25 more
  1. 2022Internal Language Model Estimation Through Explicit Context Vector Learning for Attention-based Encoder-decoder ASRcited by 9
  2. 2023JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognitioncited by 9
  3. 2022USTED: Improving ASR with a Unified Speech and Text Encoder-Decodercited by 7
  4. 2022Adaptive Discounting of Implicit Language Models in RNN-Transducerscited by 5
  5. 2022Internal Language Model Estimation based Language Model Fusion for Cross-Domain Code-Switching Speech Recognitioncited by 5
  6. 2023Text-only Domain Adaptation using Unified Speech-Text Representation in Transducercited by 5
  7. 2023On-the-fly Text Retrieval for End-to-End ASR Adaptationcited by 4
  8. 2024Alignment-Free Training for Transducer-based Multi-Talker ASRcited by 4
  9. 2022Bayesian Neural Network Language Modeling for Speech Recognitioncited by 3
  10. 2022Lattice-Free Sequence Discriminative Training for Phoneme-Based Neural Transducerscited by 3
  11. 2023Decoupled Structure for Improved Adaptability of End-to-End Modelscited by 3
  12. 2024Keep Decoding Parallel with Effective Knowledge Distillation from Language Models to End-to-end Speech Recogniserscited by 3
  13. 2024Text Injection for Neural Contextual Biasingcited by 3
  14. 2024CJST: CTC Compressor based Joint Speech and Text Training for Decoder-Only ASRcited by 3
  15. 2022An Empirical Study of Language Model Integration for Transducer based Speech Recognitioncited by 2
  16. 2023Improved Factorized Neural Transducer Model For text-only Domain Adaptationcited by 2
  17. 2025Label-Context-Dependent Internal Language Model Estimation for CTCcited by 2
  18. 2022On the Prediction Network Architecture in RNN-T for ASRcited by 1
  19. 2022Memory Augmented Lookup Dictionary based Language Modeling for Automatic Speech Recognitioncited by 1
  20. 2023Rethinking Speech Recognition with A Multimodal Perspective via Acoustic and Semantic Cooperative Decodingcited by 1
  21. 2023Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptationcited by 1
  22. 2023On the Relation between Internal Language Model and Sequence Discriminative Training for Neural Transducerscited by 1
  23. 2024Boosting Hybrid Autoregressive Transducer-based ASR with Internal Acoustic Model Training and Dual Blank Thresholdingcited by 1
  24. 2023Text-Only Domain Adaptation for End-to-End Speech Recognition through Down-Sampling Acoustic Representationcited by 0
  25. 2023Investigating the Effect of Language Models in Sequence Discriminative Training for Neural Transducerscited by 0