# Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models

arXiv: https://arxiv.org/abs/2010.01739 · HTML: https://mlsanity.com/paper/2010.01739/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2020.

## Built on by (most cited first, 12 of 12)

- [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024)
- [Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations](https://mlsanity.com/paper/2110.07581/) (2021)
- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning](https://mlsanity.com/paper/2311.11551/) (2023)
- [On the Universality of Deep Contextual Language Models](https://mlsanity.com/paper/2109.07140/) (2021)
- [BioADAPT-MRC: Adversarial Learning-based Domain Adaptation Improves Biomedical Machine Reading Comprehension Task](https://mlsanity.com/paper/2202.13174/) (2022)
- [Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking](https://mlsanity.com/paper/2203.13151/) (2022)
- [Generalizing through Forgetting -- Domain Generalization for Symptom Event Extraction in Clinical Notes](https://mlsanity.com/paper/2209.09485/) (2022)
- [Generalised Unsupervised Domain Adaptation of Neural Machine Translation with Cross-Lingual Data Selection](https://mlsanity.com/paper/2109.04292/) (2021)
- [Out-of-Distribution Generalization in Text Classification: Past, Present, and Future](https://mlsanity.com/paper/2305.14104/) (2023)
- [Extracting Biomedical Entities from Noisy Audio Transcripts](https://mlsanity.com/paper/2403.17363/) (2024)
- [Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation](https://mlsanity.com/paper/2506.15702/) (2025)

## Abstract (arXiv)

Recent work has shown the importance of adaptation of broad-coverage contextualised embedding models on the domain of the target task of interest. Current self-supervised adaptation methods are simplistic, as the training signal comes from a small percentage of randomly masked-out tokens. In this paper, we show that careful masking strategies can bridge the knowledge gap of masked language models (MLMs) about the domains more effectively by allocating self-supervision where it is needed. Furthermore, we propose an effective training strategy by adversarially masking out those tokens which are harder to reconstruct by the underlying MLM. The adversarial objective leads to a challenging combinatorial optimisation problem over subsets of tokens, which we tackle efficiently through relaxation to a variational lowerbound and dynamic programming. On six unsupervised domain adaptation tasks involving named entity recognition, our method strongly outperforms the random masking strategy and achieves up to +1.64 F1 score improvements.