# Contrastive Domain Adaptation for Question Answering using Limited Text Corpora

arXiv: https://arxiv.org/abs/2108.13854 · HTML: https://mlsanity.com/paper/2108.13854/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 59 references that are in the map.
- First posted on arXiv 31 Aug 2021.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1901.00976/) (2019)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [Synthetic QA Corpora Generation with Roundtrip Consistency](https://mlsanity.com/paper/1906.05416/) (2019)
- [The Effect of Natural Distribution Shift on Question Answering Models](https://mlsanity.com/paper/2004.14444/) (2020)
- [Training Question Answering Models From Synthetic Data](https://mlsanity.com/paper/2002.09599/) (2020)
- [Retrospective Reader for Machine Reading Comprehension](https://mlsanity.com/paper/2001.09694/) (2020)
- [End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems](https://mlsanity.com/paper/2010.06028/) (2020)
- [How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations](https://mlsanity.com/paper/1909.04925/) (2019)
- [MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension](https://mlsanity.com/paper/1905.13453/) (2019)
- [Question Answering and Question Generation as Dual Tasks](https://mlsanity.com/paper/1706.02027/) (2017)
- [Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs](https://mlsanity.com/paper/2005.13837/) (2020)
- [Simple and Effective Semi-Supervised Question Answering](https://mlsanity.com/paper/1804.00720/) (2018)
- [Domain-agnostic Question-Answering with Adversarial Training](https://mlsanity.com/paper/1910.09342/) (2019)
- [Harvesting and Refining Question-Answer Pairs for Unsupervised QA](https://mlsanity.com/paper/2005.02925/) (2020)

## Built on by (most cited first, 20 of 20)

- [AnglE-optimized Text Embeddings](https://mlsanity.com/paper/2309.12871/) (2023)
- [ConDA: Contrastive Domain Adaptation for AI-generated Text Detection](https://mlsanity.com/paper/2309.03992/) (2023)
- [Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark](https://mlsanity.com/paper/2406.01607/) (2024)
- [Contrastive Domain Adaptation for Early Misinformation Detection: A Case Study on COVID-19](https://mlsanity.com/paper/2208.09578/) (2022)
- [Simulating Bandit Learning from User Feedback for Extractive Question Answering](https://mlsanity.com/paper/2203.10079/) (2022)
- [Synthetic Question Value Estimation for Domain Adaptation of Question Answering](https://mlsanity.com/paper/2203.08926/) (2022)
- [Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracing](https://mlsanity.com/paper/2410.01727/) (2024)
- [GSQA: An End-to-End Model for Generative Spoken Question Answering](https://mlsanity.com/paper/2312.09781/) (2023)
- [Domain Adaptation for Question Answering via Question Classification](https://mlsanity.com/paper/2209.04998/) (2022)
- [Generative Data Augmentation using LLMs improves Distributional Robustness in Question Answering](https://mlsanity.com/paper/2309.06358/) (2023)
- [Unsupervised Domain Adaptation for COVID-19 Information Service with Contrastive Adversarial Domain Mixup](https://mlsanity.com/paper/2210.03250/) (2022)
- [Understanding Domain Learning in Language Models Through Subpopulation Analysis](https://mlsanity.com/paper/2210.12553/) (2022)
- [Investigating Zero- and Few-shot Generalization in Fact Verification](https://mlsanity.com/paper/2309.09444/) (2023)
- [Source-Free Domain Adaptation for Question Answering with Masked Self-training](https://mlsanity.com/paper/2212.09563/) (2022)
- [QA Domain Adaptation using Hidden Space Augmentation and Self-Supervised Contrastive Adaptation](https://mlsanity.com/paper/2210.10861/) (2022)
- [Zero- and Few-Shot Event Detection via Prompt-Based Meta Learning](https://mlsanity.com/paper/2305.17373/) (2023)
- [JCSE: Contrastive Learning of Japanese Sentence Embeddings and Its Applications](https://mlsanity.com/paper/2301.08193/) (2023)
- [DomainInv: Domain Invariant Fine Tuning and Adversarial Label Correction For QA Domain Adaptation](https://mlsanity.com/paper/2305.05589/) (2023)
- [Using Weak Supervision and Data Augmentation in Question Answering](https://mlsanity.com/paper/2309.16175/) (2023)
- [CSSL: Contrastive Self-Supervised Learning for Dependency Parsing on Relatively Free Word Ordered and Morphologically Rich Low Resource Languages](https://mlsanity.com/paper/2410.06944/) (2024)

## Abstract (arXiv)

Question generation has recently shown impressive results in customizing question answering (QA) systems to new domains. These approaches circumvent the need for manually annotated training data from the new domain and, instead, generate synthetic question-answer pairs that are used for training. However, existing methods for question generation rely on large amounts of synthetically generated datasets and costly computational resources, which render these techniques widely inaccessible when the text corpora is of limited size. This is problematic as many niche domains rely on small text corpora, which naturally restricts the amount of synthetic data that can be generated. In this paper, we propose a novel framework for domain adaptation called contrastive domain adaptation for QA (CAQA). Specifically, CAQA combines techniques from question generation and domain-invariant learning to answer out-of-domain questions in settings with limited text corpora. Here, we train a QA system on both source data and generated data from the target domain with a contrastive adaptation loss that is incorporated in the training objective. By combining techniques from question generation and domain-invariant learning, our model achieved considerable improvements compared to state-of-the-art baselines.