# Multilingual Neural Machine Translation with Language Clustering

arXiv: https://arxiv.org/abs/1908.09324 · HTML: https://mlsanity.com/paper/1908.09324/

## Key facts

- Cited by 47 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 25 Aug 2019.

## Built on by (most cited first, 30 of 47)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages](https://mlsanity.com/paper/2104.05596/) (2021)
- [Facebook AI WMT21 News Translation Task Submission](https://mlsanity.com/paper/2108.03265/) (2021)
- [xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning](https://mlsanity.com/paper/2401.07037/) (2024)
- [Counter-Interference Adapter for Multilingual Machine Translation](https://mlsanity.com/paper/2104.08154/) (2021)
- [On Generalization in Coreference Resolution](https://mlsanity.com/paper/2109.09667/) (2021)
- [Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval](https://mlsanity.com/paper/2311.05800/) (2023)
- [Importance-based Neuron Allocation for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2107.06569/) (2021)
- [Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation](https://mlsanity.com/paper/2404.11201/) (2024)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [Multilingual Machine Translation with Hyper-Adapters](https://mlsanity.com/paper/2205.10835/) (2022)
- [Language-Family Adapters for Low-Resource Multilingual Neural Machine Translation](https://mlsanity.com/paper/2209.15236/) (2022)
- [MMTAfrica: Multilingual Machine Translation for African Languages](https://mlsanity.com/paper/2204.04306/) (2022)
- [Towards Better Chinese-centric Neural Machine Translation for Low-resource Languages](https://mlsanity.com/paper/2204.04344/) (2022)
- [Multilingual Translation via Grafting Pre-trained Language Models](https://mlsanity.com/paper/2109.05256/) (2021)
- [GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation](https://mlsanity.com/paper/2207.14467/) (2022)
- [LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalization](https://mlsanity.com/paper/2401.06034/) (2024)
- [Language Embeddings Sometimes Contain Typological Generalizations](https://mlsanity.com/paper/2301.08115/) (2023)
- [Adaptive Sparse Transformer for Multilingual Translation](https://mlsanity.com/paper/2104.07358/) (2021)
- [Language Embeddings for Typology and Cross-lingual Transfer Learning](https://mlsanity.com/paper/2106.02082/) (2021)
- [Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter](https://mlsanity.com/paper/2305.12449/) (2023)
- [GradSim: Gradient-Based Language Grouping for Effective Multilingual Training](https://mlsanity.com/paper/2310.15269/) (2023)
- [Parameter Differentiation based Multilingual Neural Machine Translation](https://mlsanity.com/paper/2112.13619/) (2021)
- [Discovering Representation Sprachbund For Multilingual Pre-Training](https://mlsanity.com/paper/2109.00271/) (2021)
- [On the Universality of Deep Contextual Language Models](https://mlsanity.com/paper/2109.07140/) (2021)
- [What is Next when Sequential Prediction Meets Implicitly Hard Interaction?](https://mlsanity.com/paper/2202.06620/) (2022)
- [HLT-MT: High-resource Language-specific Training for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2207.04906/) (2022)
- [Selected Languages are All You Need for Cross-lingual Truthfulness Transfer](https://mlsanity.com/paper/2406.14434/) (2024)
- [MT4CrossOIE: Multi-stage Tuning for Cross-lingual Open Information Extraction](https://mlsanity.com/paper/2308.06552/) (2023)
- [Towards a Deep Understanding of Multilingual End-to-End Speech Translation](https://mlsanity.com/paper/2310.20456/) (2023)

## Abstract (arXiv)

Multilingual neural machine translation (NMT), which translates multiple languages using a single model, is of great practical importance due to its advantages in simplifying the training process, reducing online maintenance costs, and enhancing low-resource and zero-shot translation. Given there are thousands of languages in the world and some of them are very different, it is extremely burdensome to handle them all in a single model or use a separate model for each language pair. Therefore, given a fixed resource budget, e.g., the number of models, how to determine which languages should be supported by one model is critical to multilingual NMT, which, unfortunately, has been ignored by previous work. In this work, we develop a framework that clusters languages into different groups and trains one multilingual model for each cluster. We study two methods for language clustering: (1) using prior knowledge, where we cluster languages according to language family, and (2) using language embedding, in which we represent each language by an embedding vector and cluster them in the embedding space. In particular, we obtain the embedding vectors of all the languages by training a universal neural machine translation model. Our experiments on 23 languages show that the first clustering method is simple and easy to understand but leading to suboptimal translation accuracy, while the second method sufficiently captures the relationship among languages well and improves the translation accuracy for almost all the languages over baseline methods