# Large-Scale Domain Adaptation via Teacher-Student Learning

arXiv: https://arxiv.org/abs/1708.05466 · HTML: https://mlsanity.com/paper/1708.05466/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Aug 2017.

## Built on by (most cited first, 30 of 31)

- [Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching](https://mlsanity.com/paper/2102.02973/) (2021)
- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning](https://mlsanity.com/paper/2105.13617/) (2021)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition](https://mlsanity.com/paper/2106.02302/) (2021)
- [Listen, Adapt, Better WER: Source-free Single-utterance Test-time Adaptation for Automatic Speech Recognition](https://mlsanity.com/paper/2203.14222/) (2022)
- [A Unified Approach to Coreset Learning](https://mlsanity.com/paper/2111.03044/) (2021)
- [Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition](https://mlsanity.com/paper/2110.04934/) (2021)
- [StructVPR: Distill Structural Knowledge with Weighting Samples for Visual Place Recognition](https://mlsanity.com/paper/2212.00937/) (2022)
- [Leveraging Acoustic and Linguistic Embeddings from Pretrained speech and language Models for Intent Classification](https://mlsanity.com/paper/2102.07370/) (2021)
- [XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition](https://mlsanity.com/paper/2103.08207/) (2021)
- [Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillation](https://mlsanity.com/paper/2105.03544/) (2021)
- [Boosting Cross-Domain Speech Recognition with Self-Supervision](https://mlsanity.com/paper/2206.09783/) (2022)
- [Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition](https://mlsanity.com/paper/2303.10917/) (2023)
- [Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models](https://mlsanity.com/paper/2405.14161/) (2024)
- [Collaborative Training of Acoustic Encoders for Speech Recognition](https://mlsanity.com/paper/2106.08960/) (2021)
- [Wav2vec-S: Semi-Supervised Pre-Training for Low-Resource ASR](https://mlsanity.com/paper/2110.04484/) (2021)
- [Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greek](https://mlsanity.com/paper/2301.00304/) (2022)
- [Modular Domain Adaptation for Conformer-Based Streaming ASR](https://mlsanity.com/paper/2305.13408/) (2023)
- [Advancing Test-Time Adaptation in Wild Acoustic Test Settings](https://mlsanity.com/paper/2310.09505/) (2023)
- [StructVPR++: Distill Structural and Semantic Knowledge with Weighting Samples for Visual Place Recognition](https://mlsanity.com/paper/2503.06601/) (2025)
- [Knowledge Distillation Leveraging Alternative Soft Targets from Non-Parallel Qualified Speech Data](https://mlsanity.com/paper/2112.08878/) (2021)
- [A Comprehensive Review of Machine Learning Advances on Data Change: A Cross-Field Perspective](https://mlsanity.com/paper/2402.12627/) (2024)
- [Customizing Speech Recognition Model with Large Language Model Feedback](https://mlsanity.com/paper/2506.11091/) (2025)
- [Exploring Teacher-Student Learning Approach for Multi-lingual Speech-to-Intent Classification](https://mlsanity.com/paper/2109.13486/) (2021)
- [Sequence-level self-learning with multiple hypotheses](https://mlsanity.com/paper/2112.05826/) (2021)
- [AB/BA analysis: A framework for estimating keyword spotting recall improvement while maintaining audio privacy](https://mlsanity.com/paper/2204.08474/) (2022)
- [Speaker- and Age-Invariant Training for Child Acoustic Modeling Using Adversarial Multi-Task Learning](https://mlsanity.com/paper/2210.10231/) (2022)
- [Noise robust distillation of self-supervised speech models via correlation metrics](https://mlsanity.com/paper/2312.12153/) (2023)

## Abstract (arXiv)

High accuracy speech recognition requires a large amount of transcribed data for supervised training. In the absence of such data, domain adaptation of a well-trained acoustic model can be performed, but even here, high accuracy usually requires significant labeled data from the target domain. In this work, we propose an approach to domain adaptation that does not require transcriptions but instead uses a corpus of unlabeled parallel data, consisting of pairs of samples from the source domain of the well-trained model and the desired target domain. To perform adaptation, we employ teacher/student (T/S) learning, in which the posterior probabilities generated by the source-domain model can be used in lieu of labels to train the target-domain model. We evaluate the proposed approach in two scenarios, adapting a clean acoustic model to noisy speech and adapting an adults speech acoustic model to children speech. Significant improvements in accuracy are obtained, with reductions in word error rate of up to 44% over the original source model without the need for transcribed data in the target domain. Moreover, we show that increasing the amount of unlabeled data results in additional model robustness, which is particularly beneficial when using simulated training data in the target-domain.