# Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translation

arXiv: https://arxiv.org/abs/2212.10551 · HTML: https://mlsanity.com/paper/2212.10551/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 29 of 29)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [StableMoE: Stable Routing Strategy for Mixture of Experts](https://mlsanity.com/paper/2204.08396/) (2022)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Universal Neural Machine Translation for Extremely Low Resource Languages](https://mlsanity.com/paper/1802.05368/) (2018)
- [Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information](https://mlsanity.com/paper/2010.03142/) (2020)
- [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://mlsanity.com/paper/1610.03017/) (2016)
- [Learning Language Specific Sub-network for Multilingual Machine Translation](https://mlsanity.com/paper/2105.09259/) (2021)
- [Counter-Interference Adapter for Multilingual Machine Translation](https://mlsanity.com/paper/2104.08154/) (2021)
- [Multi-Source Neural Translation](https://mlsanity.com/paper/1601.00710/) (2016)
- [Parameter Sharing Methods for Multilingual Self-Attentional Translation Models](https://mlsanity.com/paper/1809.00252/) (2018)
- [Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/1912.01214/) (2019)
- [Robust Optimization for Multilingual Translation with Imbalanced Data](https://mlsanity.com/paper/2104.07639/) (2021)
- [Multilingual Translation via Grafting Pre-trained Language Models](https://mlsanity.com/paper/2109.05256/) (2021)
- [LegoNN: Building Modular Encoder-Decoder Models](https://mlsanity.com/paper/2206.03318/) (2022)
- [Revisiting Modularized Multilingual NMT to Meet Industrial Demands](https://mlsanity.com/paper/2010.09402/) (2020)

## Built on by (most cited first, 21 of 21)

- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages](https://mlsanity.com/paper/2407.05975/) (2024)
- [Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models](https://mlsanity.com/paper/2311.09278/) (2023)
- [A Survey of Multilingual Reasoning in Language Models](https://mlsanity.com/paper/2502.09457/) (2025)
- [How Vocabulary Sharing Facilitates Multilingualism in LLaMA?](https://mlsanity.com/paper/2311.09071/) (2023)
- [Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions](https://mlsanity.com/paper/2402.18025/) (2024)
- [AFRIDOC-MT: Document-level MT Corpus for African Languages](https://mlsanity.com/paper/2501.06374/) (2025)
- [Krey`ol-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages](https://mlsanity.com/paper/2405.05376/) (2024)
- [Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources](https://mlsanity.com/paper/2504.04152/) (2025)
- [Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model](https://mlsanity.com/paper/2403.11621/) (2024)
- [Cross-Lingual Transfer of Cultural Knowledge: An Asymmetric Phenomenon](https://mlsanity.com/paper/2506.01675/) (2025)
- [Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data](https://mlsanity.com/paper/2506.00469/) (2025)
- [KS-Lottery: Finding Certified Lottery Tickets for Multilingual Language Models](https://mlsanity.com/paper/2402.02801/) (2024)
- [Asynchronous and Segmented Bidirectional Encoding for NMT](https://mlsanity.com/paper/2402.14849/) (2024)
- [MAMMOTH: Massively Multilingual Modular Open Translation @ Helsinki](https://mlsanity.com/paper/2403.07544/) (2024)
- [Pula: Training Large Language Models for Setswana](https://mlsanity.com/paper/2408.02239/) (2024)
- [EnAnchored-X2X: English-Anchored Optimization for Many-to-Many Translation](https://mlsanity.com/paper/2509.19770/) (2025)
- [Limited-Resource Adapters Are Regularizers, Not Linguists](https://mlsanity.com/paper/2505.24525/) (2025)
- [Enhancing Multilingual Reasoning via Steerable Model Merging](https://mlsanity.com/paper/2606.19002/) (2026)
- [Drift-Constrained Optimization: Only Direction Matters in Fine-Tuning Instruct Models](https://mlsanity.com/paper/2609.13680/) (2026)
- [Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation](https://mlsanity.com/paper/2609.13770/) (2026)

## Abstract (arXiv)

Multilingual neural machine translation (MNMT) aims to build a unified model for many language directions. Existing monolithic models for MNMT encounter two challenges: parameter interference among languages and inefficient inference for large models. In this paper, we revisit the classic multi-way structures and develop a detachable model by assigning each language (or group of languages) to an individual branch that supports plug-and-play training and inference. To address the needs of learning representations for all languages in a unified space, we propose a novel efficient training recipe, upon which we build an effective detachable model, Lego-MT. For a fair comparison, we collect data from OPUS and build a translation benchmark covering 433 languages and 1.3B parallel data. Experiments show that Lego-MT with 1.2B parameters brings an average gain of 3.2 spBLEU. It even outperforms M2M-100 with 12B parameters. The proposed training recipe brings a 28.2× speedup over the conventional multi-way training method.footnote urlhttps://github.com/CONE-MT/Lego-MT.