mlsanityData to 7 Oct 2026

The Trade-offs of Domain Adaptation for Neural Language Models

arXiv 2109.10274 ↗First posted 21 Sep 2021cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

This work connects language model adaptation with concepts of machine learning theory. We consider a training setup with a large out-of-domain set and a small in-domain set. We derive how the benefit of training a model on either set depends on the size of the sets and the distance between their underlying distributions. We analyze how out-of-domain pre-training before in-domain fine-tuning achieves better generalization than either solution independently. Finally, we present how adaptation techniques based on data selection, such as importance sampling, intelligent data selection and influence functions, can be presented in a common framework which highlights their similarity and also their subtle differences.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 24 of its 52 references that are in the map. First posted on arXiv 21 Sep 2021.

Builds on

24 of 52 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Convex optimizationcited by 2,479
  9. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  11. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  12. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
12 more
  1. 2016Language Modeling with Gated Convolutional Networkscited by 688
  2. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  3. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  4. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  5. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  6. 2020Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Datacited by 50
  7. 2018Policy Optimization via Importance Samplingcited by 47
  8. 2018Denoising Neural Machine Translation Training with Trusted Data and Online Data Selectioncited by 43
  9. 2017Dynamic Data Selection for Neural Machine Translationcited by 33
  10. 2017Regularization techniques for fine-tuning in neural machine translationcited by 27
  11. 2021On the Complementarity of Data Selection and Fine Tuning for Domain Adaptationcited by 10
  12. 2021Gradient-guided Loss Masking for Neural Machine Translationcited by 8

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 6; 2024: 3; 2025: 2.

14 papers.

Sort:

By area:

  1. 2022On the Domain Adaptation and Generalization of Pretrained Language Models: A Surveycited by 22
  2. 2023In-Context Demonstration Selection with Cross Entropy Differencecited by 17
  3. 2025Towards General Continuous Memory for Vision-Language Modelscited by 14
  4. 2022Influence Functions for Sequence Tagging Modelscited by 9
  5. 2023Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarizationcited by 9
  6. 2024DALLMi: Domain Adaption for LLM-based Multi-label Classifiercited by 7
  7. 2024Need a Small Specialized Language Model? Plan Early!cited by 4
  8. 2022Generalizing through Forgetting -- Domain Generalization for Symptom Event Extraction in Clinical Notescited by 2
  9. 2023A Formalism and Approach for Improving Robustness of Large Language Models Using Risk-Adjusted Confidence Scorescited by 2
  10. 2023Domain Adaptation for Arabic Machine Translation: The Case of Financial Textscited by 1
  11. 2024Defining and Evaluating Decision and Composite Risk in Language Models Applied to Natural Language Inferencecited by 1
  12. 2025Transferable text data distillation by trajectory matchingcited by 1
2 more
  1. 2023Epi-Curriculum: Episodic Curriculum Learning for Low-Resource Domain Adaptation in Neural Machine Translationcited by 0
  2. 2023Source Prompt: Coordinated Pre-training of Language Models on Diverse Corpora from Multiple Sourcescited by 0