mlsanityData to 14 Oct 2026

UDALM: Unsupervised Domain Adaptation through Language Modeling

arXiv 2104.07078 ↗First posted 14 Apr 2021cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained language models for downstream tasks. We introduce UDALM, a fine-tuning procedure, using a mixed classification and Masked Language Model loss, that can adapt to the target domain distribution in a robust and sample efficient manner. Our experiments show that performance of models trained with the mixed loss scales with the amount of available target data and the mixed loss can be effectively used as a stopping criterion during UDA training. Furthermore, we discuss the relationship between A-distance and the target error and explore some limitations of the Domain Adversarial Training approach. Our method is evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset, yielding 91.74% accuracy, which is an 1.11% absolute improvement over the state-of-the-art.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 33 of its 61 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

33 of 61 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  11. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
21 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  2. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  3. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  4. 2016Domain Separation Networkscited by 320
  5. 2018A DIRT-T Approach to Unsupervised Domain Adaptationcited by 215
  6. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  7. 2017Asymmetric Tri-training for Unsupervised Domain Adaptationcited by 152
  8. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
  9. 2018Multi-Source Domain Adaptation with Mixture of Expertscited by 62
  10. 2018Strong Baselines for Neural Semi-supervised Learning under Domain Shiftcited by 39
  11. 2019An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Modelscited by 39
  12. 2020PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Modelscited by 33
  13. 2018Domain Adaptation with Adversarial Training and Graph Embeddingscited by 30
  14. 2018Adversarial Domain Adaptation for Duplicate Question Detectioncited by 30
  15. 2020Multi-Source Domain Adaptation for Text Classification via DistanceNet-Banditscited by 30
  16. 2019Deep Contextualized Self-training for Low Resource Dependency Parsingcited by 24
  17. 2016Neural Structural Correspondence Learning for Domain Adaptationcited by 19
  18. 2018What's in a Domain? Learning Domain-Robust Text Representations using Adversarial Trainingcited by 17
  19. 2019Simplified Neural Unsupervised Domain Adaptationcited by 10
  20. 2020Meta Fine-Tuning Neural Language Models for Multi-Domain Text Miningcited by 6
  21. 2020Feature Adaptation of Pre-Trained Language Models across Languages and Domains with Robust Self-Trainingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 11; 2023: 10; 2024: 5; 2025: 3; 2026: 1.

33 papers.

Sort:

By area:

  1. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  2. 2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 64
  3. 2022Balancing Discriminability and Transferability for Source-Free Domain Adaptationcited by 58
  4. 2022AdaPrompt: Adaptive Model Training for Prompt-based NLPcited by 30
  5. 2022On the Domain Adaptation and Generalization of Pretrained Language Models: A Surveycited by 23
  6. 2022COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learningcited by 20
  7. 2023Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learningcited by 16
  8. 2021Unsupervised Domain Adaptation with Adaptercited by 14
  9. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  10. 2023UDApter -- Efficient Domain Adaptation Using Adapterscited by 13
  11. 2023VIBE: Topic-Driven Temporal Adaptation for Twitter Classificationcited by 12
  12. 2023Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Studycited by 8
21 more
  1. 2022Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greekcited by 7
  2. 2024LLMs Could Autonomously Learn Without External Supervisioncited by 6
  3. 2023Challenges in Domain-Specific Abstractive Summarization and How to Overcome themcited by 5
  4. 2024How Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation?cited by 5
  5. 2021SentimentArcs: A Novel Method for Self-Supervised Sentiment Analysis of Time Series Shows SOTA Transformers Can Struggle Finding Narrative Arcscited by 4
  6. 2022Domain Confused Contrastive Learning for Unsupervised Domain Adaptationcited by 4
  7. 2023Improving Few-shot Generalization of Safety Classifiers via Data Augmented Parameter-Efficient Fine-Tuningcited by 4
  8. 2023A Two-Stage Framework with Self-Supervised Distillation For Cross-Domain Text Classificationcited by 3
  9. 2022Generalizing through Forgetting -- Domain Generalization for Symptom Event Extraction in Clinical Notescited by 2
  10. 2022Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gapscited by 2
  11. 2022Pivotal Role of Language Modeling in Recommender Systems: Enriching Task-specific and Task-agnostic Representation Learningcited by 2
  12. 2023Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLPcited by 2
  13. 2023TACIT: A Target-Agnostic Feature Disentanglement Framework for Cross-Domain Text Classificationcited by 2
  14. 2024Can Humans Identify Domains?cited by 2
  15. 2022Few-shot Mining of Naturally Occurring Inputs and Outputscited by 1
  16. 2024A Comprehensive Review of Machine Learning Advances on Data Change: A Cross-Field Perspectivecited by 1
  17. 2025MSDA: Combining Pseudo-labeling and Self-Supervision for Unsupervised Domain Adaptation in ASRcited by 1
  18. 2024Causal Discovery Inspired Unsupervised Domain Adaptation for Emotion-Cause Pair Extractioncited by 0
  19. 2025Aligning LLMs with Domain Invariant Reward Modelscited by 0
  20. 2025Similarity-Based Domain Adaptation with LLMscited by 0
  21. 2026MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptationcited by 0