# Building a Multi-domain Neural Machine Translation Model using Knowledge Distillation

arXiv: https://arxiv.org/abs/2004.07324 · HTML: https://mlsanity.com/paper/2004.07324/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2020.

## Built on by (most cited first, 9 of 9)

- [Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study](https://mlsanity.com/paper/2104.00676/) (2021)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?](https://mlsanity.com/paper/2206.14532/) (2022)
- [Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Models](https://mlsanity.com/paper/2210.15184/) (2022)
- [Information Redundancy and Biases in Public Document Information Extraction Benchmarks](https://mlsanity.com/paper/2304.14936/) (2023)
- [SocialBERT -- Transformers for Online SocialNetwork Language Modelling](https://mlsanity.com/paper/2111.07148/) (2021)
- [Robust Domain Adaptation for Pre-trained Multilingual Neural Machine Translation Models](https://mlsanity.com/paper/2210.14979/) (2022)
- [KD4MT: A Survey of Knowledge Distillation for Machine Translation](https://mlsanity.com/paper/2602.15845/) (2026)
- [Multi-Domain Learning From Insufficient Annotations](https://mlsanity.com/paper/2305.02757/) (2023)

## Abstract (arXiv)

Lack of specialized data makes building a multi-domain neural machine translation tool challenging. Although emerging literature dealing with low resource languages starts to show promising results, most state-of-the-art models used millions of sentences. Today, the majority of multi-domain adaptation techniques are based on complex and sophisticated architectures that are not adapted for real-world applications. So far, no scalable method is performing better than the simple yet effective mixed-finetuning, i.e finetuning a generic model with a mix of all specialized data and generic data. In this paper, we propose a new training pipeline where knowledge distillation and multiple specialized teachers allow us to efficiently finetune a model without adding new costs at inference time. Our experiments demonstrated that our training pipeline allows improving the performance of multi-domain translation over finetuning in configurations with 2, 3, and 4 domains by up to 2 points in BLEU.