# Source-Free Domain Adaptation for Question Answering with Masked Self-training

arXiv: https://arxiv.org/abs/2212.09563 · HTML: https://mlsanity.com/paper/2212.09563/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 19 Dec 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [LEEP: A New Measure to Evaluate Transferability of Learned Representations](https://mlsanity.com/paper/2002.12462/) (2020)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data](https://mlsanity.com/paper/2110.03374/) (2021)
- [Learning Discrete Representations via Information Maximizing Self-Augmented Training](https://mlsanity.com/paper/1702.08720/) (2017)
- [PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models](https://mlsanity.com/paper/2006.09075/) (2020)
- [CliniQG4QA: Generating Diverse Questions for Domain Adaptation of Clinical Question Answering](https://mlsanity.com/paper/2010.16021/) (2020)
- [Adversarial Domain Adaptation for Machine Reading Comprehension](https://mlsanity.com/paper/1908.09209/) (2019)
- [Contrastive Domain Adaptation for Question Answering using Limited Text Corpora](https://mlsanity.com/paper/2108.13854/) (2021)
- [Unsupervised Domain Adaptation of Language Models for Reading Comprehension](https://mlsanity.com/paper/1911.10768/) (2019)
- [Unsupervised Domain Adaptation on Reading Comprehension](https://mlsanity.com/paper/1911.06137/) (2019)
- [Synthetic Question Value Estimation for Domain Adaptation of Question Answering](https://mlsanity.com/paper/2203.08926/) (2022)
- [DILBERT: Customized Pre-Training for Domain Adaptation withCategory Shift, with an Application to Aspect Extraction](https://mlsanity.com/paper/2109.00571/) (2021)
- [Domain-Augmented Domain Adaptation](https://mlsanity.com/paper/2202.10000/) (2022)
- [Continual Machine Reading Comprehension via Uncertainty-aware Fixed Memory and Adversarial Domain Adaptation](https://mlsanity.com/paper/2208.05217/) (2022)

## Built on by (most cited first, 3 of 3)

- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [MABR: Multilayer Adversarial Bias Removal Without Prior Bias Knowledge](https://mlsanity.com/paper/2408.05497/) (2024)
- [Semi-supervised CAPP Transformer Learning via Pseudo-labeling](https://mlsanity.com/paper/2602.01419/) (2026)

## Abstract (arXiv)

Most previous unsupervised domain adaptation (UDA) methods for question answering(QA) require access to source domain data while fine-tuning the model for the target domain. Source domain data may, however, contain sensitive information and may be restricted. In this study, we investigate a more challenging setting, source-free UDA, in which we have only the pretrained source model and target domain data, without access to source domain data. We propose a novel self-training approach to QA models that integrates a unique mask module for domain adaptation. The mask is auto-adjusted to extract key domain knowledge while trained on the source domain. To maintain previously learned domain knowledge, certain mask weights are frozen during adaptation, while other weights are adjusted to mitigate domain shifts with pseudo-labeled samples generated in the target domain. %As part of the self-training process, we generate pseudo-labeled samples in the target domain based on models trained in the source domain. Our empirical results on four benchmark datasets suggest that our approach significantly enhances the performance of pretrained QA models on the target domain, and even outperforms models that have access to the source data during adaptation.