mlsanityData to 14 Oct 2026

Multi-Stage Pre-training for Low-Resource Domain Adaptation

arXiv 2010.05904 ↗First posted 12 Oct 2020cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Transfer learning techniques are particularly useful in NLP tasks where a sizable amount of high-quality annotated data is difficult to obtain. Current approaches directly adapt a pre-trained language model (LM) on in-domain text before fine-tuning to downstream tasks. We show that extending the vocabulary of the LM with domain-specific terms leads to further gains. To a bigger effect, we utilize structure in the unlabeled data to create auxiliary synthetic tasks, which helps the LM transfer to downstream tasks. We apply these approaches incrementally on a pre-trained Roberta-large LM and show considerable performance gain on three tasks in the IT domain: Extractive Reading Comprehension, Document Ranking and Duplicate Question Detection.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 11; 2023: 1; 2024: 4; 2025: 1; 2026: 1.

24 papers.

Sort:

By area:

  1. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  2. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  3. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 83
  4. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  5. 2021Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domainscited by 36
  6. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  7. 2022On the Domain Adaptation and Generalization of Pretrained Language Models: A Surveycited by 23
  8. 2022Representation Learning for Resource-Constrained Keyphrase Generationcited by 14
  9. 2022VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understandingcited by 12
  10. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  11. 2021Towards Robust Neural Retrieval Models with Synthetic Pre-Trainingcited by 9
  12. 2024Prompting-based Synthetic Data Generation for Few-Shot Question Answeringcited by 9
12 more
  1. 2022Adapting BigScience Multilingual Model to Unseen Languagescited by 6
  2. 2022Synthetic Target Domain Supervision for Open Retrieval QAcited by 6
  3. 2022Combining Data Generation and Active Learning for Low-Resource Question Answeringcited by 5
  4. 2024Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepalicited by 5
  5. 2022Effective Pre-Training Objectives for Transformer-based Autoencoderscited by 3
  6. 2023Decouple knowledge from parameters for plug-and-play language modelingcited by 3
  7. 2025Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languagescited by 3
  8. 2026Mind the Gap in Cultural Alignment: Task-Aware Culture Management for Large Language Modelscited by 3
  9. 2021Learning Bill Similarity with Annotated and Augmented Corpora of Billscited by 1
  10. 2022Global Readiness of Language Technology for Healthcare: What would it Take to Combat the Next Pandemic?cited by 1
  11. 2022Task Transfer and Domain Adaptation for Zero-Shot Question Answeringcited by 1
  12. 2024Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Surveycited by 1