# Breaking Down Multilingual Machine Translation

arXiv: https://arxiv.org/abs/2110.08130 · HTML: https://mlsanity.com/paper/2110.08130/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 15 Oct 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Do Attention Heads in BERT Track Syntactic Dependencies?](https://mlsanity.com/paper/1911.12246/) (2019)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Investigating Multilingual NMT Representations at Scale](https://mlsanity.com/paper/1909.02197/) (2019)
- [Universal Neural Machine Translation for Extremely Low Resource Languages](https://mlsanity.com/paper/1802.05368/) (2018)
- [Parameter Sharing Methods for Multilingual Self-Attentional Translation Models](https://mlsanity.com/paper/1809.00252/) (2018)
- [Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures](https://mlsanity.com/paper/1808.08946/) (2018)
- [Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture](https://mlsanity.com/paper/2005.03454/) (2020)
- [Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations](https://mlsanity.com/paper/2004.14923/) (2020)
- [Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation](https://mlsanity.com/paper/1809.05218/) (2018)

## Built on by (most cited first, 8 of 8)

- [Automating Code Review Activities by Large-Scale Pre-training](https://mlsanity.com/paper/2203.09095/) (2022)
- [On the Robustness of Aspect-based Sentiment Analysis: Rethinking Model, Data, and Training](https://mlsanity.com/paper/2304.09563/) (2023)
- [Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translation](https://mlsanity.com/paper/2305.14016/) (2023)
- [Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation](https://mlsanity.com/paper/2211.01292/) (2022)
- [Disentangling the Roles of Target-Side Transfer and Regularization in Multilingual Machine Translation](https://mlsanity.com/paper/2402.01772/) (2024)
- [Leveraging Reward Models for Guiding Code Review Comment Generation](https://mlsanity.com/paper/2506.04464/) (2025)
- [MetaXLR -- Mixed Language Meta Representation Transformation for Low-resource Cross-lingual Learning based on Multi-Armed Bandit](https://mlsanity.com/paper/2306.00100/) (2023)
- [NLIP_Lab-IITH Low-Resource MT System for WMT24 Indic MT Shared Task](https://mlsanity.com/paper/2410.03215/) (2024)

## Abstract (arXiv)

While multilingual training is now an essential ingredient in machine translation (MT) systems, recent work has demonstrated that it has different effects in different multilingual settings, such as many-to-one, one-to-many, and many-to-many learning. These training settings expose the encoder and the decoder in a machine translation model with different data distributions. In this paper, we examine how different varieties of multilingual training contribute to learning these two components of the MT model. Specifically, we compare bilingual models with encoders and/or decoders initialized by multilingual training. We show that multilingual training is beneficial to encoders in general, while it only benefits decoders for low-resource languages (LRLs). We further find the important attention heads for each language pair and compare their correlations during inference. Our analysis sheds light on how multilingual translation models work and enables us to propose methods to improve performance by training with highly related languages. Our many-to-one models for high-resource languages and one-to-many models for LRL outperform the best results reported by Aharoni et al. (2019)