# Domain Adversarial Fine-Tuning as an Effective Regularizer

arXiv: https://arxiv.org/abs/2009.13366 · HTML: https://mlsanity.com/paper/2009.13366/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2020.

## Built on by (most cited first, 8 of 8)

- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations](https://mlsanity.com/paper/2110.07581/) (2021)
- [Invariant Language Modeling](https://mlsanity.com/paper/2110.08413/) (2021)
- [Learning List-Level Domain-Invariant Representations for Ranking](https://mlsanity.com/paper/2212.10764/) (2022)
- [Domain Adaptation in Multilingual and Multi-Domain Monolingual Settings for Complex Word Identification](https://mlsanity.com/paper/2205.07283/) (2022)
- [Out-of-Distribution Generalization in Text Classification: Past, Present, and Future](https://mlsanity.com/paper/2305.14104/) (2023)
- [To Tune or Not To Tune? Zero-shot Models for Legal Case Entailment](https://mlsanity.com/paper/2202.03120/) (2022)
- [DELIA: Diversity-Enhanced Learning for Instruction Adaptation in Large Language Models](https://mlsanity.com/paper/2408.10841/) (2024)

## Abstract (arXiv)

In Natural Language Processing (NLP), pretrained language models (LMs) that are transferred to downstream tasks have been recently shown to achieve state-of-the-art results. However, standard fine-tuning can degrade the general-domain representations captured during pretraining. To address this issue, we introduce a new regularization technique, AFTER; domain Adversarial Fine-Tuning as an Effective Regularizer. Specifically, we complement the task-specific loss used during fine-tuning with an adversarial objective. This additional loss term is related to an adversarial classifier, that aims to discriminate between in-domain and out-of-domain text representations. In-domain refers to the labeled dataset of the task at hand while out-of-domain refers to unlabeled data from a different domain. Intuitively, the adversarial classifier acts as a regularizer which prevents the model from overfitting to the task-specific domain. Empirical results on various natural language understanding tasks show that AFTER leads to improved performance compared to standard fine-tuning.