# Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learning

arXiv: https://arxiv.org/abs/2110.00165 · HTML: https://mlsanity.com/paper/2110.00165/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Generative Pre-Training for Speech with Autoregressive Predictive Coding](https://mlsanity.com/paper/1910.12607/) (2019)
- [Recognizing long-form speech using streaming end-to-end models](https://mlsanity.com/paper/1910.11455/) (2019)
- [A Better and Faster End-to-End Model for Streaming ASR](https://mlsanity.com/paper/2011.10798/) (2020)
- [Toward domain-invariant speech recognition via large scale training](https://mlsanity.com/paper/1808.05312/) (2018)
- [Learning Word-Level Confidence For Subword End-to-End ASR](https://mlsanity.com/paper/2103.06716/) (2021)
- [Contrastive Semi-supervised Learning for ASR](https://mlsanity.com/paper/2103.05149/) (2021)

## Built on by (most cited first, 26 of 26)

- [Joint Unsupervised and Supervised Training for Multilingual ASR](https://mlsanity.com/paper/2111.08137/) (2021)
- [Pseudo Label Is Better Than Human Label](https://mlsanity.com/paper/2203.12668/) (2022)
- [Anatomy of Industrial Scale Multilingual ASR](https://mlsanity.com/paper/2404.09841/) (2024)
- [Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASR](https://mlsanity.com/paper/2210.10027/) (2022)
- [Efficient Domain Adaptation for Speech Foundation Models](https://mlsanity.com/paper/2302.01496/) (2023)
- [Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition](https://mlsanity.com/paper/2309.15223/) (2023)
- [PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations](https://mlsanity.com/paper/2203.16965/) (2022)
- [Boosting Cross-Domain Speech Recognition with Self-Supervision](https://mlsanity.com/paper/2206.09783/) (2022)
- [Improving Deliberation by Text-Only and Semi-Supervised Training](https://mlsanity.com/paper/2206.14716/) (2022)
- [Production federated keyword spotting via distillation, filtering, and joint federated-centralized training](https://mlsanity.com/paper/2204.06322/) (2022)
- [Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models](https://mlsanity.com/paper/2405.14161/) (2024)
- [Wav2vec-S: Semi-Supervised Pre-Training for Low-Resource ASR](https://mlsanity.com/paper/2110.04484/) (2021)
- [Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greek](https://mlsanity.com/paper/2301.00304/) (2022)
- [Large vocabulary speech recognition for languages of Africa: multilingual modeling and self-supervised learning](https://mlsanity.com/paper/2208.03067/) (2022)
- [Semi-Self-Supervised Domain Adaptation: Developing Deep Learning Models with Limited Annotated Data for Wheat Head Segmentation](https://mlsanity.com/paper/2405.07157/) (2024)
- [Confidence-based Ensembles of End-to-End Speech Recognition Models](https://mlsanity.com/paper/2306.15824/) (2023)
- [Resource-Efficient Transfer Learning From Speech Foundation Model Using Hierarchical Feature Fusion](https://mlsanity.com/paper/2211.02712/) (2022)
- [Better Semi-supervised Learning for Multi-domain ASR Through Incremental Retraining and Data Filtering](https://mlsanity.com/paper/2506.04981/) (2025)
- [BEST-RQ-Based Self-Supervised Learning for Whisper Domain Adaptation](https://mlsanity.com/paper/2510.24570/) (2025)
- [Damage Control During Domain Adaptation for Transducer Based Automatic Speech Recognition](https://mlsanity.com/paper/2210.03255/) (2022)
- [MSDA: Combining Pseudo-labeling and Self-Supervision for Unsupervised Domain Adaptation in ASR](https://mlsanity.com/paper/2505.24656/) (2025)
- [Customizing Speech Recognition Model with Large Language Model Feedback](https://mlsanity.com/paper/2506.11091/) (2025)
- [Gumbel-BEARD: Automatic Layer Selection for Self-Supervised Adaptation of Whisper in Low-Resource Domains](https://mlsanity.com/paper/2606.11429/) (2026)
- [Feature Normalization for Fine-tuning Self-Supervised Models in Speech Enhancement](https://mlsanity.com/paper/2306.08406/) (2023)
- [LESS: Large Language Model Enhanced Semi-Supervised Learning for Speech Foundational Models Using in-the-wild Data](https://mlsanity.com/paper/2506.04586/) (2025)
- [Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR](https://mlsanity.com/paper/2604.06487/) (2026)

## Abstract (arXiv)

Self- and semi-supervised learning methods have been actively investigated to reduce labeled training data or enhance the model performance. However, the approach mostly focus on in-domain performance for public datasets. In this study, we utilize the combination of self- and semi-supervised learning methods to solve unseen domain adaptation problem in a large-scale production setting for online ASR model. This approach demonstrates that using the source domain data with a small fraction of the target domain data (3%) can recover the performance gap compared to a full data baseline: relative 13.5% WER improvement for target domain data.