# Back-Training excels Self-Training at Unsupervised Domain Adaptation of Question Generation and Passage Retrieval

arXiv: https://arxiv.org/abs/2104.08801 · HTML: https://mlsanity.com/paper/2104.08801/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 63 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [Universal Domain Adaptation through Self Supervision](https://mlsanity.com/paper/2002.07953/) (2020)
- [ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection](https://mlsanity.com/paper/2103.05346/) (2021)
- [Neural Unsupervised Domain Adaptation in NLP---A Survey](https://mlsanity.com/paper/2006.00632/) (2020)
- [Synthetic QA Corpora Generation with Roundtrip Consistency](https://mlsanity.com/paper/1906.05416/) (2019)
- [Self-Training for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.09116/) (2019)
- [Controllable Abstractive Summarization](https://mlsanity.com/paper/1711.05217/) (2017)
- [Generating Natural Questions About an Image](https://mlsanity.com/paper/1603.06059/) (2016)
- [Unsupervised Statistical Machine Translation](https://mlsanity.com/paper/1809.01272/) (2018)
- [Regularizing Neural Machine Translation by Target-bidirectional Agreement](https://mlsanity.com/paper/1808.04064/) (2018)
- [Question Answering and Question Generation as Dual Tasks](https://mlsanity.com/paper/1706.02027/) (2017)
- [Self-Training for End-to-End Speech Translation](https://mlsanity.com/paper/2006.02490/) (2020)
- [Supervised and Unsupervised Transfer Learning for Question Answering](https://mlsanity.com/paper/1711.05345/) (2017)
- [Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds](https://mlsanity.com/paper/1911.02365/) (2019)
- [A Joint Model for Question Answering and Question Generation](https://mlsanity.com/paper/1706.01450/) (2017)
- [Structured Training for Neural Network Transition-Based Parsing](https://mlsanity.com/paper/1506.06158/) (2015)
- [End-to-End QA on COVID-19: Domain Adaptation with Synthetic Training](https://mlsanity.com/paper/2012.01414/) (2020)

## Built on by (most cited first, 8 of 8)

- [PeerQA: A Scientific Question Answering Dataset from Peer Reviews](https://mlsanity.com/paper/2502.13668/) (2025)
- [DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine](https://mlsanity.com/paper/2203.10232/) (2022)
- [Simulating Bandit Learning from User Feedback for Extractive Question Answering](https://mlsanity.com/paper/2203.10079/) (2022)
- [Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey](https://mlsanity.com/paper/2208.03197/) (2022)
- [Retrieval Augmentation for Commonsense Reasoning: A Unified Approach](https://mlsanity.com/paper/2210.12887/) (2022)
- [Few-shot Question Generation for Personalized Feedback in Intelligent Tutoring Systems](https://mlsanity.com/paper/2206.04187/) (2022)
- [A New Era: Intelligent Tutoring Systems Will Transform Online Learning for Millions](https://mlsanity.com/paper/2203.03724/) (2022)
- [Domain-Expanded ASTE: Rethinking Generalization in Aspect Sentiment Triplet Extraction](https://mlsanity.com/paper/2305.14434/) (2023)

## Abstract (arXiv)

In this work, we introduce back-training, an alternative to self-training for unsupervised domain adaptation (UDA) from source to target domain. While self-training generates synthetic training data where natural inputs are aligned with noisy outputs, back-training results in natural outputs aligned with noisy inputs. This significantly reduces the gap between the target domain and synthetic data distribution, and reduces model overfitting to the source domain. We run UDA experiments on question generation and passage retrieval from the textitNatural Questions domain to machine learning and biomedical domains. We find that back-training vastly outperforms self-training by a mean improvement of 7.8 BLEU-4 points on generation, and 17.6% top-20 retrieval accuracy across both domains. We further propose consistency filters to remove low-quality synthetic data before training. We also release a new domain-adaptation dataset- textitMLQuestions containing 35K unaligned questions, 50K unaligned passages, and 3K aligned question-passage pairs.