# Causes and Cures for Interference in Multilingual Translation

arXiv: https://arxiv.org/abs/2212.07530 · HTML: https://mlsanity.com/paper/2212.07530/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 14 Dec 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://mlsanity.com/paper/2105.00572/) (2021)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Balancing Training for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2004.06748/) (2020)
- [In Defense of the Unitary Scalarization for Deep Multi-Task Learning](https://mlsanity.com/paper/2201.04122/) (2022)
- [Do Current Multi-Task Optimization Methods in Deep Learning Even Help?](https://mlsanity.com/paper/2209.11379/) (2022)
- [Data Scaling Laws in NMT: The Effect of Noise and Architecture](https://mlsanity.com/paper/2202.01994/) (2022)
- [Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2005.04816/) (2020)
- [Learning Language Specific Sub-network for Multilingual Machine Translation](https://mlsanity.com/paper/2105.09259/) (2021)
- [Counter-Interference Adapter for Multilingual Machine Translation](https://mlsanity.com/paper/2104.08154/) (2021)
- [Complete Multilingual Neural Machine Translation](https://mlsanity.com/paper/2010.10239/) (2020)
- [Adaptive Scheduling for Multi-Task Learning](https://mlsanity.com/paper/1909.06434/) (2019)
- [Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits](https://mlsanity.com/paper/2110.06997/) (2021)

## Built on by (most cited first, 23 of 23)

- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Scaling Laws for Generative Mixed-Modal Language Models](https://mlsanity.com/paper/2301.03728/) (2023)
- [Multilingual Instruction Tuning With Just a Pinch of Multilinguality](https://mlsanity.com/paper/2401.01854/) (2024)
- [From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Prediction](https://mlsanity.com/paper/2310.16802/) (2023)
- [Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?](https://mlsanity.com/paper/2312.12683/) (2023)
- [Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metric](https://mlsanity.com/paper/2402.06900/) (2024)
- [Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer](https://mlsanity.com/paper/2305.14280/) (2023)
- [Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation](https://mlsanity.com/paper/2404.11201/) (2024)
- [AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers](https://mlsanity.com/paper/2411.11188/) (2024)
- [Vintix: Action Model via In-Context Reinforcement Learning](https://mlsanity.com/paper/2501.19400/) (2025)
- [On the Pareto Front of Multilingual Neural Machine Translation](https://mlsanity.com/paper/2304.03216/) (2023)
- [Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets](https://mlsanity.com/paper/2410.04579/) (2024)
- [Conditional [MASK] Discrete Diffusion Language Model](https://mlsanity.com/paper/2411.06438/) (2024)
- [Condensing Multilingual Knowledge with Lightweight Language-Specific Modules](https://mlsanity.com/paper/2305.13993/) (2023)
- [Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Model](https://mlsanity.com/paper/2309.13018/) (2023)
- [A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation](https://mlsanity.com/paper/2403.20157/) (2024)
- [Disentangling the Roles of Target-Side Transfer and Regularization in Multilingual Machine Translation](https://mlsanity.com/paper/2402.01772/) (2024)
- [Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP](https://mlsanity.com/paper/2605.13436/) (2026)
- [Exploring Representational Disparities Between Multilingual and Bilingual Translation Models](https://mlsanity.com/paper/2305.14230/) (2023)
- [Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning](https://mlsanity.com/paper/2505.10347/) (2025)
- [LLaVA-NeuMT: Selective Layer-Neuron Modulation for Efficient Multilingual Multimodal Translation](https://mlsanity.com/paper/2507.18940/) (2025)
- [The Impact of Vocabulary Overlaps on Knowledge Transfer in Multilingual Machine Translation](https://mlsanity.com/paper/2605.04196/) (2026)
- [There is No Theoretical Curse of Multilinguality For Embedding Space Structure](https://mlsanity.com/paper/2608.17088/) (2026)

## Abstract (arXiv)

Multilingual machine translation models can benefit from synergy between different language pairs, but also suffer from interference. While there is a growing number of sophisticated methods that aim to eliminate interference, our understanding of interference as a phenomenon is still limited. This work identifies the main factors that contribute to interference in multilingual machine translation. Through systematic experimentation, we find that interference (or synergy) are primarily determined by model size, data size, and the proportion of each language pair within the total dataset. We observe that substantial interference occurs mainly when the model is very small with respect to the available training data, and that using standard transformer configurations with less than one billion parameters largely alleviates interference and promotes synergy. Moreover, we show that tuning the sampling temperature to control the proportion of each language pair in the data is key to balancing the amount of interference between low and high resource language pairs effectively, and can lead to superior performance overall.