# UDApter -- Efficient Domain Adaptation Using Adapters

arXiv: https://arxiv.org/abs/2302.03194 · HTML: https://mlsanity.com/paper/2302.03194/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 7 Feb 2023.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Domain Separation Networks](https://mlsanity.com/paper/1608.06019/) (2016)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning](https://mlsanity.com/paper/1702.08811/) (2017)
- [Correlation Alignment for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1612.01939/) (2016)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://mlsanity.com/paper/2106.03164/) (2021)
- [Co-regularized Alignment for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1811.05443/) (2018)
- [Robust Transfer Learning with Pretrained Language Models through Adapters](https://mlsanity.com/paper/2108.02340/) (2021)
- [UDALM: Unsupervised Domain Adaptation through Language Modeling](https://mlsanity.com/paper/2104.07078/) (2021)
- [Continual Sequence Generation with Adaptive Compositional Modules](https://mlsanity.com/paper/2203.10652/) (2022)
- [Domain Divergences: a Survey and Empirical Analysis](https://mlsanity.com/paper/2010.12198/) (2020)
- [Unsupervised Domain Adaptation with Adapter](https://mlsanity.com/paper/2111.00667/) (2021)
- [Learning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension](https://mlsanity.com/paper/2204.00996/) (2022)

## Built on by (most cited first, 13 of 13)

- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain](https://mlsanity.com/paper/2307.03042/) (2023)
- [Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost](https://mlsanity.com/paper/2306.15766/) (2023)
- [Decoding Time Series with LLMs: A Multi-Agent Framework for Cross-Domain Annotation](https://mlsanity.com/paper/2410.17462/) (2024)
- [AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation](https://mlsanity.com/paper/2503.19693/) (2025)
- [RE-Adapt: Reverse Engineered Adaptation of Large Language Models](https://mlsanity.com/paper/2405.15007/) (2024)
- [CATS: Mitigating Correlation Shift for Multivariate Time Series Classification](https://mlsanity.com/paper/2504.04283/) (2025)
- [How Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation?](https://mlsanity.com/paper/2401.17514/) (2024)
- [DAdEE: Unsupervised Domain Adaptation in Early Exit PLMs](https://mlsanity.com/paper/2410.04424/) (2024)
- [What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition](https://mlsanity.com/paper/2401.12756/) (2024)
- [Multi-Task Domain Adaptation for Language Grounding with 3D Objects](https://mlsanity.com/paper/2407.02846/) (2024)
- [Fixed and Adaptive Simultaneous Machine Translation Strategies Using Adapters](https://mlsanity.com/paper/2407.13469/) (2024)
- [MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2606.22272/) (2026)

## Abstract (arXiv)

We propose two methods to make unsupervised domain adaptation (UDA) more parameter efficient using adapters, small bottleneck layers interspersed with every layer of the large-scale pre-trained language model (PLM). The first method deconstructs UDA into a two-step process: first by adding a domain adapter to learn domain-invariant information and then by adding a task adapter that uses domain-invariant information to learn task representations in the source domain. The second method jointly learns a supervised classifier while reducing the divergence measure. Compared to strong baselines, our simple methods perform well in natural language inference (MNLI) and the cross-domain sentiment classification task. We even outperform unsupervised domain adaptation methods such as DANN and DSN in sentiment classification, and we are within 0.85% F1 for natural language inference task, by fine-tuning only a fraction of the full model parameters. We release our code at https://github.com/declare-lab/domadapter