# Towards domain generalisation in ASR with elitist sampling and ensemble knowledge distillation

arXiv: https://arxiv.org/abs/2303.00550 · HTML: https://mlsanity.com/paper/2303.00550/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 15 of 15)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Distilling the Knowledge of BERT for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2008.03822/) (2020)
- [Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data](https://mlsanity.com/paper/2110.15094/) (2021)
- [Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation](https://mlsanity.com/paper/2107.02629/) (2021)
- [Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2001.01798/) (2020)
- [Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognition](https://mlsanity.com/paper/2005.09310/) (2020)
- [Domain Generalization on Efficient Acoustic Scene Classification using Residual Normalization](https://mlsanity.com/paper/2111.06531/) (2021)
- [Domain Adaptation Using Class Similarity for Robust Speech Recognition](https://mlsanity.com/paper/2011.02782/) (2020)
- [Insights on Neural Representations for End-to-End Speech Recognition](https://mlsanity.com/paper/2205.09456/) (2022)

## Built on by (most cited first, 2 of 2)

- [Progressive unsupervised domain adaptation for ASR using ensemble models and multi-stage training](https://mlsanity.com/paper/2402.04805/) (2024)
- [MUST: A Multilingual Student-Teacher Learning approach for low-resource speech recognition](https://mlsanity.com/paper/2310.18865/) (2023)

## Abstract (arXiv)

Knowledge distillation has widely been used for model compression and domain adaptation for speech applications. In the presence of multiple teachers, knowledge can easily be transferred to the student by averaging the models output. However, previous research shows that the student do not adapt well with such combination. This paper propose to use an elitist sampling strategy at the output of ensemble teacher models to select the best-decoded utterance generated by completely out-of-domain teacher models for generalizing unseen domain. The teacher models are trained on AMI, LibriSpeech and WSJ while the student is adapted for the Switchboard data. The results show that with the selection strategy based on the individual models posteriors the student model achieves a better WER compared to all the teachers and baselines with a minimum absolute improvement of about 8.4 percent. Furthermore, an insights on the model adaptation with out-of-domain data has also been studied via correlation analysis.