# Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/2001.01798 · HTML: https://mlsanity.com/paper/2001.01798/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jan 2020.

## Built on by (most cited first, 25 of 25)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2102.01380/) (2021)
- [On Addressing Practical Challenges for RNN-Transducer](https://mlsanity.com/paper/2105.00858/) (2021)
- [Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition](https://mlsanity.com/paper/2106.02302/) (2021)
- [Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.05354/) (2021)
- [Modular Hybrid Autoregressive Transducer](https://mlsanity.com/paper/2210.17049/) (2022)
- [Layer Pruning on Demand with Intermediate CTC](https://mlsanity.com/paper/2106.09216/) (2021)
- [XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition](https://mlsanity.com/paper/2103.08207/) (2021)
- [Collaborative Training of Acoustic Encoders for Speech Recognition](https://mlsanity.com/paper/2106.08960/) (2021)
- [Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation](https://mlsanity.com/paper/2211.00508/) (2022)
- [Unsupervised Domain Expansion for Visual Categorization](https://mlsanity.com/paper/2104.00233/) (2021)
- [Progressive unsupervised domain adaptation for ASR using ensemble models and multi-stage training](https://mlsanity.com/paper/2402.04805/) (2024)
- [Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Data](https://mlsanity.com/paper/2305.15971/) (2023)
- [Towards domain generalisation in ASR with elitist sampling and ensemble knowledge distillation](https://mlsanity.com/paper/2303.00550/) (2023)
- [Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition](https://mlsanity.com/paper/2312.12783/) (2023)
- [Audio Enhancement for Computer Audition -- An Iterative Training Paradigm Using Sample Importance](https://mlsanity.com/paper/2408.06264/) (2024)
- [Knowledge Distillation Leveraging Alternative Soft Targets from Non-Parallel Qualified Speech Data](https://mlsanity.com/paper/2112.08878/) (2021)
- [Black-box Generalization of Machine Teaching](https://mlsanity.com/paper/2206.15205/) (2022)
- [Sequence-level self-learning with multiple hypotheses](https://mlsanity.com/paper/2112.05826/) (2021)
- [Domain Adapting Deep Reinforcement Learning for Real-world Speech Emotion Recognition](https://mlsanity.com/paper/2207.12248/) (2022)
- [Mutual Learning of Single- and Multi-Channel End-to-End Neural Diarization](https://mlsanity.com/paper/2210.03459/) (2022)
- [Investigating Cross-Domain Behaviors of BERT in Review Understanding](https://mlsanity.com/paper/2306.15123/) (2023)
- [Self-Train Before You Transcribe](https://mlsanity.com/paper/2406.12937/) (2024)
- [Measuring Robustness of Speech Recognition from MEG Signals Under Distribution Shift](https://mlsanity.com/paper/2604.04129/) (2026)
- [Benchmarking_Fast_Domain_Adaptation_for_Unsupervised_Speech_Units](https://mlsanity.com/paper/2608.26992/) (2026)

## Abstract (arXiv)

Teacher-student (T/S) has shown to be effective for domain adaptation of deep neural network acoustic models in hybrid speech recognition systems. In this work, we extend the T/S learning to large-scale unsupervised domain adaptation of an attention-based end-to-end (E2E) model through two levels of knowledge transfer: teacher's token posteriors as soft labels and one-best predictions as decoder guidance. To further improve T/S learning with the help of ground-truth labels, we propose adaptive T/S (AT/S) learning. Instead of conditionally choosing from either the teacher's soft token posteriors or the one-hot ground-truth label, in AT/S, the student always learns from both the teacher and the ground truth with a pair of adaptive weights assigned to the soft and one-hot labels quantifying the confidence on each of the knowledge sources. The confidence scores are dynamically estimated at each decoder step as a function of the soft and one-hot labels. With 3400 hours parallel close-talk and far-field Microsoft Cortana data for domain adaptation, T/S and AT/S achieve 6.3% and 10.3% relative word error rate improvement over a strong E2E model trained with the same amount of far-field data.