# Unsupervised Domain Adaptation of Language Models for Reading Comprehension

arXiv: https://arxiv.org/abs/1911.10768 · HTML: https://mlsanity.com/paper/1911.10768/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2019.

## Built on by (most cited first, 15 of 15)

- [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024)
- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges](https://mlsanity.com/paper/2405.15604/) (2024)
- [Prompting-based Synthetic Data Generation for Few-Shot Question Answering](https://mlsanity.com/paper/2405.09335/) (2024)
- [Simple or Complex? Complexity-Controllable Question Generation with Soft Templates and Deep Mixture of Experts Model](https://mlsanity.com/paper/2110.06560/) (2021)
- [Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering Data](https://mlsanity.com/paper/2102.01226/) (2021)
- [Combining Data Generation and Active Learning for Low-Resource Question Answering](https://mlsanity.com/paper/2211.14880/) (2022)
- [Improving Few-shot Generalization of Safety Classifiers via Data Augmented Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2310.16959/) (2023)
- [A Semi-supervised Learning Approach with Two Teachers to Improve Breakdown Identification in Dialogues](https://mlsanity.com/paper/2202.10948/) (2022)
- [BioADAPT-MRC: Adversarial Learning-based Domain Adaptation Improves Biomedical Machine Reading Comprehension Task](https://mlsanity.com/paper/2202.13174/) (2022)
- [Source-Free Domain Adaptation for Question Answering with Masked Self-training](https://mlsanity.com/paper/2212.09563/) (2022)
- [Question answering systems for health professionals at the point of care -- a systematic review](https://mlsanity.com/paper/2402.01700/) (2024)
- [Zero-Shot Estimation of Base Models' Weights in Ensemble of Machine Reading Comprehension Systems for Robust Generalization](https://mlsanity.com/paper/2106.16013/) (2021)
- [Zero-shot Entity and Tweet Characterization with Designed Conditional Prompts and Contexts](https://mlsanity.com/paper/2204.08405/) (2022)
- [Characterizing Latent Perspectives of Media Houses Towards Public Figures](https://mlsanity.com/paper/2309.06112/) (2023)

## Abstract (arXiv)

This study tackles unsupervised domain adaptation of reading comprehension (UDARC). Reading comprehension (RC) is a task to learn the capability for question answering with textual sources. State-of-the-art models on RC still do not have general linguistic intelligence; i.e., their accuracy worsens for out-domain datasets that are not used in the training. We hypothesize that this discrepancy is caused by a lack of the language modeling (LM) capability for the out-domain. The UDARC task allows models to use supervised RC training data in the source domain and only unlabeled passages in the target domain. To solve the UDARC problem, we provide two domain adaptation models. The first one learns the out-domain LM and in-domain RC task sequentially. The second one is the proposed model that uses a multi-task learning approach of LM and RC. The models can retain both the RC capability acquired from the supervised data in the source domain and the LM capability from the unlabeled data in the target domain. We evaluated the models on UDARC with five datasets in different domains. The models outperformed the model without domain adaptation. In particular, the proposed model yielded an improvement of 4.3/4.2 points in EM/F1 in an unseen biomedical domain.