# Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters

arXiv: https://arxiv.org/abs/2110.09574 · HTML: https://mlsanity.com/paper/2110.09574/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 53 references that are in the map.
- First posted on arXiv 18 Oct 2021.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [AdapterDrop: On the Efficiency of Adapters in Transformers](https://mlsanity.com/paper/2010.11918/) (2020)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [Unsupervised Domain Clusters in Pretrained Language Models](https://mlsanity.com/paper/2004.02105/) (2020)

## Built on by (most cited first, 21 of 21)

- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models](https://mlsanity.com/paper/2302.07027/) (2023)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations](https://mlsanity.com/paper/2305.14224/) (2023)
- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation](https://mlsanity.com/paper/2503.19693/) (2025)
- [NAVER LABS Europe's Multilingual Speech Translation Systems for the IWSLT 2023 Low-Resource Track](https://mlsanity.com/paper/2306.07763/) (2023)
- [Multilingual Pre-training with Language and Task Adaptation for Multilingual Text Style Transfer](https://mlsanity.com/paper/2203.08552/) (2022)
- [When does Parameter-Efficient Transfer Learning Work for Machine Translation?](https://mlsanity.com/paper/2205.11277/) (2022)
- [Local Byte Fusion for Neural Machine Translation](https://mlsanity.com/paper/2205.11490/) (2022)
- [M2QA: Multi-domain Multilingual Question Answering](https://mlsanity.com/paper/2407.01091/) (2024)
- [Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey](https://mlsanity.com/paper/2510.12178/) (2025)
- [What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition](https://mlsanity.com/paper/2401.12756/) (2024)
- [A Systematic Analysis of Vocabulary and BPE Settings for Optimal Fine-tuning of NMT: A Case Study of In-domain Translation](https://mlsanity.com/paper/2303.00722/) (2023)
- [m4Adapter: Multilingual Multi-Domain Adaptation for Machine Translation with a Meta-Adapter](https://mlsanity.com/paper/2210.11912/) (2022)
- [How Transferable are Attribute Controllers on Pretrained Multilingual Translation Models?](https://mlsanity.com/paper/2309.08565/) (2023)
- [Can Domains Be Transferred Across Languages in Multi-Domain Multilingual Neural Machine Translation?](https://mlsanity.com/paper/2210.11628/) (2022)
- [Vernacular Search Query Translation with Unsupervised Domain Adaptation](https://mlsanity.com/paper/2208.03711/) (2022)
- [Soft Alignment Objectives for Robust Adaptation of Language Generation](https://mlsanity.com/paper/2211.16550/) (2022)
- [Defining Boundaries: The Impact of Domain Specification on Cross-Language and Cross-Domain Transfer in Machine Translation](https://mlsanity.com/paper/2408.11926/) (2024)
- [Parameter-Efficient Fine-Tuning of Large Pretrained Models for Instance Segmentation Tasks](https://mlsanity.com/paper/2606.01947/) (2026)

## Abstract (arXiv)

Adapter layers are lightweight, learnable units inserted between transformer layers. Recent work explores using such layers for neural machine translation (NMT), to adapt pre-trained models to new domains or language pairs, training only a small set of parameters for each new setting (language pair or domain). In this work we study the compositionality of language and domain adapters in the context of Machine Translation. We aim to study, 1) parameter-efficient adaptation to multiple domains and languages simultaneously (full-resource scenario) and 2) cross-lingual transfer in domains where parallel data is unavailable for certain language pairs (partial-resource scenario). We find that in the partial resource scenario a naive combination of domain-specific and language-specific adapters often results in `catastrophic forgetting' of the missing languages. We study other ways to combine the adapters to alleviate this issue and maximize cross-lingual transfer. With our best adapter combinations, we obtain improvements of 3-4 BLEU on average for source languages that do not have in-domain data. For target languages without in-domain data, we achieve a similar improvement by combining adapters with back-translation. Supplementary material is available at https://tinyurl.com/r66stbxj