mlsanityData to 14 Oct 2026

Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition

arXiv 2110.05354 ↗First posted 6 Oct 2021cs.CL cs.AI cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Text-only adaptation of an end-to-end (E2E) model remains a challenging task for automatic speech recognition (ASR). Language model (LM) fusion-based approaches require an additional external LM during inference, significantly increasing the computation cost. To overcome this, we propose an internal LM adaptation (ILMA) of the E2E model using text-only data. Trained with audio-transcript pairs, an E2E model implicitly learns an internal LM that characterizes the token sequence probability which is approximated by the E2E model output after zeroing out the encoder contribution. During ILMA, we fine-tune the internal LM, i.e., the E2E components excluding the encoder, to minimize a cross-entropy loss. To make ILMA effective, it is essential to train the E2E model with an internal LM loss besides the standard E2E loss. Furthermore, we propose to regularize ILMA by minimizing the Kullback-Leibler divergence between the output distributions of the adapted and unadapted internal LMs. ILMA is the most effective when we update only the last linear layer of the joint network. ILMA enables a fast text-only adaptation of the E2E model without increasing the run-time computational cost. Experimented with 30K-hour trained transformer transducer models, ILMA achieves up to 34.9% relative word error rate reduction from the unadapted baseline.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

37 of 56 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2015Attention-Based Models for Speech Recognitioncited by 401
  4. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  5. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  6. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 211
  7. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  8. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 150
  9. 2020Hybrid Autoregressive Transducer (hat)cited by 108
  10. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 100
  11. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 97
  12. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 88
25 more
  1. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
  2. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  3. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 63
  4. 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 63
  5. 2021Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognitioncited by 37
  6. 2019Conditional Teacher-Student Learningcited by 34
  7. 2021Factorized Neural Transducer for Efficient Language Model Adaptationcited by 32
  8. 2018Simple Fusion: Return of the Language Modelcited by 30
  9. 2018Adversarial Teacher-Student Learning for Unsupervised Domain Adaptationcited by 28
  10. 2021Librispeech Transducer Model with Internal Language Model Prior Correctioncited by 26
  11. 2020Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognitioncited by 25
  12. 2021Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Modelscited by 25
  13. 2020Improving Tail Performance of a Deliberation E2E ASR Model Using a Large Text Corpuscited by 24
  14. 2018Speaker-Invariant Training via Adversarial Learningcited by 23
  15. 2019Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognitioncited by 23
  16. 2021Fast Text-Only Domain Adaptation of RNN-Transducer Prediction Networkcited by 19
  17. 2021Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognitioncited by 19
  18. 2018Advancing Acoustic-to-Word CTC Modelcited by 15
  19. 2016Invariant Representations for Noisy Speech Recognitioncited by 12
  20. 2019Speaker Adaptation for Attention-Based End-to-End Speech Recognitioncited by 12
  21. 2017Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognitioncited by 10
  22. 2017Deep Long Short-Term Memory Adaptive Beamforming Networks For Multichannel Robust Speech Recognitioncited by 10
  23. 2019Adversarial Speaker Adaptationcited by 10
  24. 2019Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Textcited by 10
  25. 2020L-Vector: Neural Label Embedding for Domain Adaptationcited by 9

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 3; 2023: 10; 2024: 2.

17 papers.

Sort:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  2. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  3. 2021On Language Model Integration for RNN Transducer based Speech Recognitioncited by 17
  4. 2022Modular Hybrid Autoregressive Transducercited by 17
  5. 2023Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuningcited by 13
  6. 2023JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognitioncited by 9
  7. 2023Understanding Shared Speech-Text Representationscited by 6
  8. 2022Relaxed Attention for Transformer Modelscited by 5
  9. 2023Text-only Domain Adaptation using Unified Speech-Text Representation in Transducercited by 5
  10. 2023Corpus Synthesis for Zero-shot ASR domain Adaptation using Large Language Modelscited by 5
  11. 2024Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuningcited by 4
  12. 2023RASR2: The RWTH ASR Toolkit for Generic Sequence-to-sequence Speech Recognitioncited by 3
5 more
  1. 2023Decoupled Structure for Improved Adaptability of End-to-End Modelscited by 3
  2. 2024Text Injection for Neural Contextual Biasingcited by 3
  3. 2023Augmenting text for spoken language understanding with Large Language Modelscited by 2
  4. 2022On the Prediction Network Architecture in RNN-T for ASRcited by 1
  5. 2023Text Injection for Capitalization and Turn-Taking Prediction in Speech Modelscited by 1