# UM4: Unified Multilingual Multiple Teacher-Student Model for Zero-Resource Neural Machine Translation

arXiv: https://arxiv.org/abs/2207.04900 · HTML: https://mlsanity.com/paper/2207.04900/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 11 Jul 2022.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Multilingual Neural Machine Translation with Knowledge Distillation](https://mlsanity.com/paper/1902.10461/) (2019)
- [Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation](https://mlsanity.com/paper/2104.08771/) (2021)
- [Zero-Resource Translation with Multi-Lingual Neural Machine Translation](https://mlsanity.com/paper/1606.04164/) (2016)
- [Multi-task Learning for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2010.02523/) (2020)
- [Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages](https://mlsanity.com/paper/1909.09524/) (2019)
- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [Neural Machine Translation with Pivot Languages](https://mlsanity.com/paper/1611.04928/) (2016)
- [A Multilingual View of Unsupervised Machine Translation](https://mlsanity.com/paper/2002.02955/) (2020)
- [A Teacher-Student Framework for Zero-Resource Neural Machine Translation](https://mlsanity.com/paper/1705.00753/) (2017)
- [Distributionally Robust Multilingual Machine Translation](https://mlsanity.com/paper/2109.04020/) (2021)
- [Multilingual Neural Machine Translation for Zero-Resource Languages](https://mlsanity.com/paper/1909.07342/) (2019)

## Built on by (most cited first, 10 of 10)

- [LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation](https://mlsanity.com/paper/2210.15461/) (2022)
- [GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator](https://mlsanity.com/paper/2212.10218/) (2022)
- [Lemur: Log Parsing with Entropy Sampling and Chain-of-Thought Merging](https://mlsanity.com/paper/2402.18205/) (2024)
- [GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Ranking](https://mlsanity.com/paper/2305.18144/) (2023)
- [LogLG: Weakly Supervised Log Anomaly Detection via Log-Event Graph Construction](https://mlsanity.com/paper/2208.10833/) (2022)
- [Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies](https://mlsanity.com/paper/2402.09282/) (2024)
- [HanoiT: Enhancing Context-aware Translation via Selective Context](https://mlsanity.com/paper/2301.06825/) (2023)
- [Multilingual Entity and Relation Extraction from Unified to Language-specific Training](https://mlsanity.com/paper/2301.04434/) (2023)
- [MT4CrossOIE: Multi-stage Tuning for Cross-lingual Open Information Extraction](https://mlsanity.com/paper/2308.06552/) (2023)
- [RoNID: New Intent Discovery with Generated-Reliable Labels and Cluster-friendly Representations](https://mlsanity.com/paper/2404.08977/) (2024)

## Abstract (arXiv)

Most translation tasks among languages belong to the zero-resource translation problem where parallel corpora are unavailable. Multilingual neural machine translation (MNMT) enables one-pass translation using shared semantic space for all languages compared to the two-pass pivot translation but often underperforms the pivot-based method. In this paper, we propose a novel method, named as Unified Multilingual Multiple teacher-student Model for NMT (UM4). Our method unifies source-teacher, target-teacher, and pivot-teacher models to guide the student model for the zero-resource translation. The source teacher and target teacher force the student to learn the direct source to target translation by the distilled knowledge on both source and target sides. The monolingual corpus is further leveraged by the pivot-teacher model to enhance the student model. Experimental results demonstrate that our model of 72 directions significantly outperforms previous methods on the WMT benchmark.