# Mask The Bias: Improving Domain-Adaptive Generalization of CTC-based ASR with Internal Language Model Estimation

arXiv: https://arxiv.org/abs/2305.03837 · HTML: https://mlsanity.com/paper/2305.03837/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 5 May 2023.

## Builds on (most cited first, 20 of 20)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition](https://mlsanity.com/paper/1807.10857/) (2018)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [Why does CTC result in peaky behavior?](https://mlsanity.com/paper/2105.14849/) (2021)
- [Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Models](https://mlsanity.com/paper/2104.05544/) (2021)
- [CASS-NAT: CTC Alignment-based Single Step Non-autoregressive Transformer for Speech Recognition](https://mlsanity.com/paper/2010.14725/) (2020)
- [On Language Model Integration for RNN Transducer based Speech Recognition](https://mlsanity.com/paper/2110.06841/) (2021)
- [Domain-aware Neural Language Models for Speech Recognition](https://mlsanity.com/paper/2101.03229/) (2021)
- [Adapting Long Context NLM for ASR Rescoring in Conversational Agents](https://mlsanity.com/paper/2104.11070/) (2021)
- [Benchmarking LF-MMI, CTC and RNN-T Criteria for Streaming ASR](https://mlsanity.com/paper/2011.04785/) (2020)
- [Internal Language Model Estimation Through Explicit Context Vector Learning for Attention-based Encoder-decoder ASR](https://mlsanity.com/paper/2201.11627/) (2022)
- [ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modeling](https://mlsanity.com/paper/2106.09532/) (2021)
- [A Likelihood Ratio based Domain Adaptation Method for E2E Models](https://mlsanity.com/paper/2201.03655/) (2022)

## Built on by (most cited first, 1 of 1)

- [Label-Context-Dependent Internal Language Model Estimation for CTC](https://mlsanity.com/paper/2506.06096/) (2025)

## Abstract (arXiv)

End-to-end ASR models trained on large amount of data tend to be implicitly biased towards language semantics of the training data. Internal language model estimation (ILME) has been proposed to mitigate this bias for autoregressive models such as attention-based encoder-decoder and RNN-T. Typically, ILME is performed by modularizing the acoustic and language components of the model architecture, and eliminating the acoustic input to perform log-linear interpolation with the text-only posterior. However, for CTC-based ASR, it is not as straightforward to decouple the model into such acoustic and language components, as CTC log-posteriors are computed in a non-autoregressive manner. In this work, we propose a novel ILME technique for CTC-based ASR models. Our method iteratively masks the audio timesteps to estimate a pseudo log-likelihood of the internal LM by accumulating log-posteriors for only the masked timesteps. Extensive evaluation across multiple out-of-domain datasets reveals that the proposed approach improves WER by up to 9.8% and OOV F1-score by up to 24.6% relative to Shallow Fusion, when only text data from target domain is available. In the case of zero-shot domain adaptation, with no access to any target domain data, we demonstrate that removing the source domain bias with ILME can still outperform Shallow Fusion to improve WER by up to 9.3% relative.