# Progressive unsupervised domain adaptation for ASR using ensemble models and multi-stage training

arXiv: https://arxiv.org/abs/2402.04805 · HTML: https://mlsanity.com/paper/2402.04805/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 13 of 13)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Iterative Pseudo-Labeling for Speech Recognition](https://mlsanity.com/paper/2005.09267/) (2020)
- [Rethinking Evaluation in ASR: Are Our Models Robust Enough?](https://mlsanity.com/paper/2010.11745/) (2020)
- [Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training](https://mlsanity.com/paper/2011.13439/) (2020)
- [Exploiting Adapters for Cross-lingual Low-resource Speech Recognition](https://mlsanity.com/paper/2105.11905/) (2021)
- [Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2001.01798/) (2020)
- [Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition](https://mlsanity.com/paper/2210.00169/) (2022)
- [Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognition](https://mlsanity.com/paper/2005.09310/) (2020)
- [Domain Generalization on Efficient Acoustic Scene Classification using Residual Normalization](https://mlsanity.com/paper/2111.06531/) (2021)
- [Domain Adaptation Using Class Similarity for Robust Speech Recognition](https://mlsanity.com/paper/2011.02782/) (2020)
- [Towards domain generalisation in ASR with elitist sampling and ensemble knowledge distillation](https://mlsanity.com/paper/2303.00550/) (2023)

## Built on by (most cited first, 4 of 4)

- [SAGE: Spliced-Audio Generated Data for Enhancing Foundational Models in Low-Resource Arabic-English Code-Switched Speech Recognition](https://mlsanity.com/paper/2506.22143/) (2025)
- [MSDA: Combining Pseudo-labeling and Self-Supervision for Unsupervised Domain Adaptation in ASR](https://mlsanity.com/paper/2505.24656/) (2025)
- [Pseudo2Real: Task Arithmetic for Pseudo-Label Correction in Automatic Speech Recognition](https://mlsanity.com/paper/2510.08047/) (2025)
- [On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin](https://mlsanity.com/paper/2608.11898/) (2026)

## Abstract (arXiv)

In Automatic Speech Recognition (ASR), teacher-student (T/S) training has shown to perform well for domain adaptation with small amount of training data. However, adaption without ground-truth labels is still challenging. A previous study has shown the effectiveness of using ensemble teacher models in T/S training for unsupervised domain adaptation (UDA) but its performance still lags behind compared to the model trained on in-domain data. This paper proposes a method to yield better UDA by training multi-stage students with ensemble teacher models. Initially, multiple teacher models are trained on labelled data from read and meeting domains. These teachers are used to train a student model on unlabelled out-of-domain telephone speech data. To improve the adaptation, subsequent student models are trained sequentially considering previously trained model as their teacher. Experiments are conducted with three teachers trained on AMI, WSJ and LibriSpeech and three stages of students on SwitchBoard data. Results shown on eval00 test set show significant WER improvement with multi-stage training with an absolute gain of 9.8%, 7.7% and 3.3% at each stage.