# Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey

arXiv: https://arxiv.org/abs/2104.06951 · HTML: https://mlsanity.com/paper/2104.06951/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 190 of its 367 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 30 of 190)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Descent-to-Delete: Gradient-Based Methods for Machine Unlearning](https://mlsanity.com/paper/2007.02923/) (2020)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)

## Built on by (most cited first, 30 of 46)

- [Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2308.09388/) (2023)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2305.06360/) (2023)
- [Federated Domain Generalization: A Survey](https://mlsanity.com/paper/2306.01334/) (2023)
- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [Chunk-based Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2205.12230/) (2022)
- [Large Language Model for Multi-Domain Translation: Benchmarking and Domain CoT Fine-tuning](https://mlsanity.com/paper/2410.02631/) (2024)
- [Improving Automated Program Repair with Domain Adaptation](https://mlsanity.com/paper/2212.11414/) (2022)
- [Domain-adaptive Message Passing Graph Neural Network](https://mlsanity.com/paper/2308.16470/) (2023)
- [Supervised Learning and Large Language Model Benchmarks on Mental Health Datasets: Cognitive Distortions and Suicidal Risks in Chinese Social Media](https://mlsanity.com/paper/2309.03564/) (2023)
- [Adaptor: Objective-Centric Adaptation Framework for Language Models](https://mlsanity.com/paper/2203.03989/) (2022)
- [Efficient Machine Translation Domain Adaptation](https://mlsanity.com/paper/2204.12608/) (2022)
- [When does Parameter-Efficient Transfer Learning Work for Machine Translation?](https://mlsanity.com/paper/2205.11277/) (2022)
- [Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling](https://mlsanity.com/paper/2106.10840/) (2021)
- [Low-resource Machine Translation: what for? who for? An observational study on a dedicated Tetun language translation service](https://mlsanity.com/paper/2411.12262/) (2024)
- [Selecting Parallel In-domain Sentences for Neural Machine Translation Using Monolingual Texts](https://mlsanity.com/paper/2112.06096/) (2021)
- [Tailoring Domain Adaptation for Machine Translation Quality Estimation](https://mlsanity.com/paper/2304.08891/) (2023)
- [Adaptation Speed Analysis for Fairness-aware Causal Models](https://mlsanity.com/paper/2308.16879/) (2023)
- [What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition](https://mlsanity.com/paper/2401.12756/) (2024)
- [Specification-Aware Machine Translation and Evaluation for Purpose Alignment](https://mlsanity.com/paper/2509.17559/) (2025)
- [Domain Adaptation for Time series Transformers using One-step fine-tuning](https://mlsanity.com/paper/2401.06524/) (2024)
- [QAEncoder: Towards Aligned Representation Learning in Question Answering Systems](https://mlsanity.com/paper/2409.20434/) (2024)
- [Scalable Multi-Domain Adaptation of Language Models using Modular Experts](https://mlsanity.com/paper/2410.10181/) (2024)
- [SHIFT15M: Fashion-specific dataset for set-to-set matching with several distribution shifts](https://mlsanity.com/paper/2108.12992/) (2021)
- [Can Humans Identify Domains?](https://mlsanity.com/paper/2404.01785/) (2024)
- [On Instruction-Finetuning Neural Machine Translation Models](https://mlsanity.com/paper/2410.05553/) (2024)
- [Domain adapted machine translation: What does catastrophic forgetting forget and why?](https://mlsanity.com/paper/2412.17537/) (2024)
- [KD4MT: A Survey of Knowledge Distillation for Machine Translation](https://mlsanity.com/paper/2602.15845/) (2026)
- [Domain Adaptation for Arabic Machine Translation: The Case of Financial Texts](https://mlsanity.com/paper/2309.12863/) (2023)
- [A Comprehensive Review of Machine Learning Advances on Data Change: A Cross-Field Perspective](https://mlsanity.com/paper/2402.12627/) (2024)

## Abstract (arXiv)

The development of deep learning techniques has allowed Neural Machine Translation (NMT) models to become extremely powerful, given sufficient training data and training time. However, systems struggle when translating text from a new domain with a distinct style or vocabulary. Fine-tuning on in-domain data allows good domain adaptation, but requires sufficient relevant bilingual data. Even if this is available, simple fine-tuning can cause overfitting to new data and `catastrophic forgetting' of previously learned behaviour. We concentrate on robust approaches to domain adaptation for NMT, particularly where a system may need to translate across multiple domains. We divide techniques into those revolving around data selection or generation, model architecture, parameter adaptation procedure, and inference procedure. We finally highlight the benefits of domain adaptation and multi-domain adaptation techniques to other lines of NMT research.