# Boosting Cross-Domain Speech Recognition with Self-Supervision

arXiv: https://arxiv.org/abs/2206.09783 · HTML: https://mlsanity.com/paper/2206.09783/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 99 references that are in the map.
- First posted on arXiv 20 Jun 2022.

## Builds on (most cited first, 30 of 67)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2003.03773/) (2020)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [Self-Training for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.09116/) (2019)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation](https://mlsanity.com/paper/2106.04732/) (2021)

## Built on by (most cited first, 10 of 10)

- [Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models](https://mlsanity.com/paper/2405.14161/) (2024)
- [Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greek](https://mlsanity.com/paper/2301.00304/) (2022)
- [Alternative Pseudo-Labeling for Semi-Supervised Automatic Speech Recognition](https://mlsanity.com/paper/2308.06547/) (2023)
- [Better Semi-supervised Learning for Multi-domain ASR Through Incremental Retraining and Data Filtering](https://mlsanity.com/paper/2506.04981/) (2025)
- [CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning](https://mlsanity.com/paper/2506.17818/) (2025)
- [MSDA: Combining Pseudo-labeling and Self-Supervision for Unsupervised Domain Adaptation in ASR](https://mlsanity.com/paper/2505.24656/) (2025)
- [Customizing Speech Recognition Model with Large Language Model Feedback](https://mlsanity.com/paper/2506.11091/) (2025)
- [Combining X-Vectors and Bayesian Batch Active Learning: Two-Stage Active Learning Pipeline for Speech Recognition](https://mlsanity.com/paper/2406.02566/) (2024)
- [Transliterated Zero-Shot Domain Adaptation for Automatic Speech Recognition](https://mlsanity.com/paper/2412.11185/) (2024)
- [Comparing Self-Supervised and Domain-Invariant Features for Cross-Domain Voice Phishing Detection](https://mlsanity.com/paper/2609.07079/) (2026)

## Abstract (arXiv)

The cross-domain performance of automatic speech recognition (ASR) could be severely hampered due to the mismatch between training and testing distributions. Since the target domain usually lacks labeled data, and domain shifts exist at acoustic and linguistic levels, it is challenging to perform unsupervised domain adaptation (UDA) for ASR. Previous work has shown that self-supervised learning (SSL) or pseudo-labeling (PL) is effective in UDA by exploiting the self-supervisions of unlabeled data. However, these self-supervisions also face performance degradation in mismatched domain distributions, which previous work fails to address. This work presents a systematic UDA framework to fully utilize the unlabeled data with self-supervision in the pre-training and fine-tuning paradigm. On the one hand, we apply continued pre-training and data replay techniques to mitigate the domain mismatch of the SSL pre-trained model. On the other hand, we propose a domain-adaptive fine-tuning approach based on the PL technique with three unique modifications: Firstly, we design a dual-branch PL method to decrease the sensitivity to the erroneous pseudo-labels; Secondly, we devise an uncertainty-aware confidence filtering strategy to improve pseudo-label correctness; Thirdly, we introduce a two-step PL approach to incorporate target domain linguistic knowledge, thus generating more accurate target domain pseudo-labels. Experimental results on various cross-domain scenarios demonstrate that the proposed approach effectively boosts the cross-domain performance and significantly outperforms previous approaches.