# Task Transfer and Domain Adaptation for Zero-Shot Question Answering

arXiv: https://arxiv.org/abs/2206.06705 · HTML: https://mlsanity.com/paper/2206.06705/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 19 of 19)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [CORD-19: The COVID-19 Open Research Dataset](https://mlsanity.com/paper/2004.10706/) (2020)
- [Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification](https://mlsanity.com/paper/1908.11860/) (2019)
- [Neural Domain Adaptation for Biomedical Question Answering](https://mlsanity.com/paper/1706.03610/) (2017)
- [Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension](https://mlsanity.com/paper/1911.00202/) (2019)
- [Multi-Stage Pre-training for Low-Resource Domain Adaptation](https://mlsanity.com/paper/2010.05904/) (2020)
- [Sequential Domain Adaptation through Elastic Weight Consolidation for Sentiment Analysis](https://mlsanity.com/paper/2007.01189/) (2020)
- [End-to-End QA on COVID-19: Domain Adaptation with Synthetic Training](https://mlsanity.com/paper/2012.01414/) (2020)
- [Towards Domain Adaptation from Limited Data for Question Answering Using Deep Neural Networks](https://mlsanity.com/paper/1911.02655/) (2019)
- [What Are People Asking About COVID-19? A Question Classification Dataset](https://mlsanity.com/paper/2005.12522/) (2020)
- [AdapterNet - learning input transformation for domain adaptation](https://mlsanity.com/paper/1805.11601/) (2018)

## Built on by (most cited first, 1 of 1)

- [Compositional Zero-Shot Domain Transfer with Text-to-Text Models](https://mlsanity.com/paper/2303.13386/) (2023)

## Abstract (arXiv)

Pretrained language models have shown success in various areas of natural language processing, including reading comprehension tasks. However, when applying machine learning methods to new domains, labeled data may not always be available. To address this, we use supervised pretraining on source-domain data to reduce sample complexity on domain-specific downstream tasks. We evaluate zero-shot performance on domain-specific reading comprehension tasks by combining task transfer with domain adaptation to fine-tune a pretrained model with no labelled data from the target task. Our approach outperforms Domain-Adaptive Pretraining on downstream domain-specific reading comprehension tasks in 3 out of 4 domains.