# Multi-Stage Pre-training for Low-Resource Domain Adaptation

arXiv: https://arxiv.org/abs/2010.05904 · HTML: https://mlsanity.com/paper/2010.05904/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2020.

## Built on by (most cited first, 24 of 24)

- [BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting](https://mlsanity.com/paper/2212.09535/) (2022)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Continual Pre-Training Mitigates Forgetting in Language and Vision](https://mlsanity.com/paper/2205.09357/) (2022)
- [How to Adapt Your Pretrained Multilingual Model to 1600 Languages](https://mlsanity.com/paper/2106.02124/) (2021)
- [Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains](https://mlsanity.com/paper/2106.13474/) (2021)
- [Efficient Domain Adaptation of Language Models via Adaptive Tokenization](https://mlsanity.com/paper/2109.07460/) (2021)
- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [Representation Learning for Resource-Constrained Keyphrase Generation](https://mlsanity.com/paper/2203.08118/) (2022)
- [VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding](https://mlsanity.com/paper/2211.00430/) (2022)
- [Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcare](https://mlsanity.com/paper/2405.06784/) (2024)
- [Towards Robust Neural Retrieval Models with Synthetic Pre-Training](https://mlsanity.com/paper/2104.07800/) (2021)
- [Prompting-based Synthetic Data Generation for Few-Shot Question Answering](https://mlsanity.com/paper/2405.09335/) (2024)
- [Adapting BigScience Multilingual Model to Unseen Languages](https://mlsanity.com/paper/2204.04873/) (2022)
- [Synthetic Target Domain Supervision for Open Retrieval QA](https://mlsanity.com/paper/2204.09248/) (2022)
- [Combining Data Generation and Active Learning for Low-Resource Question Answering](https://mlsanity.com/paper/2211.14880/) (2022)
- [Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepali](https://mlsanity.com/paper/2412.13860/) (2024)
- [Effective Pre-Training Objectives for Transformer-based Autoencoders](https://mlsanity.com/paper/2210.13536/) (2022)
- [Decouple knowledge from parameters for plug-and-play language modeling](https://mlsanity.com/paper/2305.11564/) (2023)
- [Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages](https://mlsanity.com/paper/2502.10140/) (2025)
- [Mind the Gap in Cultural Alignment: Task-Aware Culture Management for Large Language Models](https://mlsanity.com/paper/2602.22475/) (2026)
- [Learning Bill Similarity with Annotated and Augmented Corpora of Bills](https://mlsanity.com/paper/2109.06527/) (2021)
- [Global Readiness of Language Technology for Healthcare: What would it Take to Combat the Next Pandemic?](https://mlsanity.com/paper/2204.02790/) (2022)
- [Task Transfer and Domain Adaptation for Zero-Shot Question Answering](https://mlsanity.com/paper/2206.06705/) (2022)
- [Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey](https://mlsanity.com/paper/2411.09166/) (2024)

## Abstract (arXiv)

Transfer learning techniques are particularly useful in NLP tasks where a sizable amount of high-quality annotated data is difficult to obtain. Current approaches directly adapt a pre-trained language model (LM) on in-domain text before fine-tuning to downstream tasks. We show that extending the vocabulary of the LM with domain-specific terms leads to further gains. To a bigger effect, we utilize structure in the unlabeled data to create auxiliary synthetic tasks, which helps the LM transfer to downstream tasks. We apply these approaches incrementally on a pre-trained Roberta-large LM and show considerable performance gain on three tasks in the IT domain: Extractive Reading Comprehension, Document Ranking and Duplicate Question Detection.