# Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation

arXiv: https://arxiv.org/abs/2004.10171 · HTML: https://mlsanity.com/paper/2004.10171/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2020.

## Built on by (most cited first, 19 of 19)

- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages](https://mlsanity.com/paper/2402.12204/) (2024)
- [Multilingual Unsupervised Neural Machine Translation with Denoising Adapters](https://mlsanity.com/paper/2110.10472/) (2021)
- [Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation](https://mlsanity.com/paper/2112.03473/) (2021)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation](https://mlsanity.com/paper/2404.14827/) (2024)
- [Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2305.08096/) (2023)
- [You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models](https://mlsanity.com/paper/2210.07135/) (2022)
- [Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model](https://mlsanity.com/paper/2211.01200/) (2022)
- [Multi-Teacher Knowledge Distillation For Text Image Machine Translation](https://mlsanity.com/paper/2305.05226/) (2023)
- [Can Multilinguality benefit Non-autoregressive Machine Translation?](https://mlsanity.com/paper/2112.08570/) (2021)
- [Efficient Sub-structured Knowledge Distillation](https://mlsanity.com/paper/2203.04825/) (2022)
- [MALM: Mixing Augmented Language Modeling for Zero-Shot Machine Translation](https://mlsanity.com/paper/2210.00320/) (2022)
- [Dialogue State Distillation Network with Inter-slot Contrastive Learning for Dialogue State Tracking](https://mlsanity.com/paper/2302.08220/) (2023)
- [Can depth-adaptive BERT perform better on binary classification tasks](https://mlsanity.com/paper/2111.10951/) (2021)
- [One-stop Training of Multiple Capacity Models](https://mlsanity.com/paper/2305.14066/) (2023)
- [HITSZ's End-To-End Speech Translation Systems Combining Sequence-to-Sequence Auto Speech Recognition Model and Indic Large Language Model for IWSLT 2025 in Indic Track](https://mlsanity.com/paper/2507.19616/) (2025)
- [Unsupervised Cross-Lingual Part-of-Speech Tagging with Monolingual Corpora Only](https://mlsanity.com/paper/2602.09366/) (2026)
- [DualAnchor: Preserving Language Priors and Improving Lexical Fidelity in Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2607.27614/) (2026)

## Abstract (arXiv)

Unsupervised neural machine translation (UNMT) has recently achieved remarkable results for several language pairs. However, it can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time. That is, research on multilingual UNMT has been limited. In this paper, we empirically introduce a simple method to translate between thirteen languages using a single encoder and a single decoder, making use of multilingual data to improve UNMT for all language pairs. On the basis of the empirical findings, we propose two knowledge distillation methods to further enhance multilingual UNMT performance. Our experiments on a dataset with English translated to and from twelve other languages (including three language families and six language branches) show remarkable results, surpassing strong unsupervised individual baselines while achieving promising performance between non-English language pairs in zero-shot translation scenarios and alleviating poor performance in low-resource language pairs.