# Adaptable End-to-End ASR Models using Replaceable Internal LMs and Residual Softmax

arXiv: https://arxiv.org/abs/2302.08579 · HTML: https://mlsanity.com/paper/2302.08579/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2023.

## Built on by (most cited first, 8 of 8)

- [Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition](https://mlsanity.com/paper/2306.16007/) (2023)
- [ZeroPrompt: Streaming Acoustic Encoders are Zero-Shot Masked LMs](https://mlsanity.com/paper/2305.10649/) (2023)
- [Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning](https://mlsanity.com/paper/2307.10274/) (2023)
- [Corpus Synthesis for Zero-shot ASR domain Adaptation using Large Language Models](https://mlsanity.com/paper/2309.10707/) (2023)
- [Decoupled Structure for Improved Adaptability of End-to-End Models](https://mlsanity.com/paper/2308.13345/) (2023)
- [Improved Factorized Neural Transducer Model For text-only Domain Adaptation](https://mlsanity.com/paper/2309.09524/) (2023)
- [Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptation](https://mlsanity.com/paper/2309.07369/) (2023)
- [WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformers](https://mlsanity.com/paper/2509.10452/) (2025)

## Abstract (arXiv)

End-to-end (E2E) automatic speech recognition (ASR) implicitly learns the token sequence distribution of paired audio-transcript training data. However, it still suffers from domain shifts from training to testing, and domain adaptation is still challenging. To alleviate this problem, this paper designs a replaceable internal language model (RILM) method, which makes it feasible to directly replace the internal language model (LM) of E2E ASR models with a target-domain LM in the decoding stage when a domain shift is encountered. Furthermore, this paper proposes a residual softmax (R-softmax) that is designed for CTC-based E2E ASR models to adapt to the target domain without re-training during inference. For E2E ASR models trained on the LibriSpeech corpus, experiments showed that the proposed methods gave a 2.6% absolute WER reduction on the Switchboard data and a 1.0% WER reduction on the AESRC2020 corpus while maintaining intra-domain ASR results.