mlsanityData to 14 Oct 2026

Label-Context-Dependent Internal Language Model Estimation for CTC

arXiv 2506.06096 ↗First posted 6 Jun 2025cs.SD cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Although connectionist temporal classification (CTC) has the label context independence assumption, it can still implicitly learn a context-dependent internal language model (ILM) due to modern powerful encoders. In this work, we investigate the implicit context dependency modeled in the ILM of CTC. To this end, we propose novel context-dependent ILM estimation methods for CTC based on knowledge distillation (KD) with theoretical justifications. Furthermore, we introduce two regularization methods for KD. We conduct experiments on Librispeech and TED-LIUM Release 2 datasets for in-domain and cross-domain evaluation, respectively. Experimental results show that context-dependent ILMs outperform the context-independent priors in cross-domain evaluation, indicating that CTC learns a context-dependent ILM. The proposed label-level KD with smoothing method surpasses other ILM estimation approaches, with more than 13% relative improvement in word error rate compared to shallow fusion.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 13 of its 26 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

13 of 26 references.

References · 13

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2019Language Modeling with Deep Transformerscited by 96
  4. 2015EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decodingcited by 73
  5. 2021Advancing RNN Transducer Technology for Speech Recognitioncited by 44
  6. 2021Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognitioncited by 37
  7. 2020Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboardcited by 36
  8. 2021Librispeech Transducer Model with Internal Language Model Prior Correctioncited by 26
  9. 2021Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Modelscited by 25
  10. 2021On Language Model Integration for RNN Transducer based Speech Recognitioncited by 17
  11. 2024Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognitioncited by 7
  12. 2023Mask The Bias: Improving Domain-Adaptive Generalization of CTC-based ASR with Internal Language Model Estimationcited by 1
1 more
  1. 2023On the Relation between Internal Language Model and Sequence Discriminative Training for Neural Transducerscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Building Tailored Speech Recognizers for Japanese Speaking Assessmentcited by 1
  2. 2026Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Inter-KD: Intermediate Knowledge Distillation for CTC-Based Automatic Speech Recognition
  2. 2021Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models
  3. 2023Self-distillation Regularized Connectionist Temporal Classification Loss for Text Recognition: A Simple Yet Effective Approach
  4. 2024CR-CTC: Consistency regularization on CTC for improved speech recognition
  5. 2025Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspective
  6. 2024Efficient LLM Context Distillation
  7. 2025Analyzing the Importance of Blank for CTC-Based Knowledge Distillation
  8. 2023Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective