# Distributionally Robust Multilingual Machine Translation

arXiv: https://arxiv.org/abs/2109.04020 · HTML: https://mlsanity.com/paper/2109.04020/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach](https://mlsanity.com/paper/1610.03425/) (2016)
- [Large-Scale Methods for Distributionally Robust Optimization](https://mlsanity.com/paper/2010.05893/) (2020)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [Variance-based regularization with convex objectives](https://mlsanity.com/paper/1610.02581/) (2016)
- [Distributionally Robust Language Modeling](https://mlsanity.com/paper/1909.02060/) (2019)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Data-Driven Robust Optimization](https://mlsanity.com/paper/1401.0212/) (2013)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Balancing Training for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2004.06748/) (2020)
- [On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment](https://mlsanity.com/paper/2010.03017/) (2020)

## Built on by (most cited first, 18 of 18)

- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages](https://mlsanity.com/paper/2305.18098/) (2023)
- [Re-weighting Based Group Fairness Regularization via Classwise Robust Optimization](https://mlsanity.com/paper/2303.00442/) (2023)
- [Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting](https://mlsanity.com/paper/2210.05404/) (2022)
- [CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation](https://mlsanity.com/paper/2210.07022/) (2022)
- [Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning](https://mlsanity.com/paper/2210.14867/) (2022)
- [Distributionally Robust Optimization via Ball Oracle Acceleration](https://mlsanity.com/paper/2203.13225/) (2022)
- [Tencent's Multilingual Machine Translation System for WMT22 Large-Scale African Languages](https://mlsanity.com/paper/2210.09644/) (2022)
- [UM4: Unified Multilingual Multiple Teacher-Student Model for Zero-Resource Neural Machine Translation](https://mlsanity.com/paper/2207.04900/) (2022)
- [Towards Higher Pareto Frontier in Multilingual Machine Translation](https://mlsanity.com/paper/2305.15718/) (2023)
- [On the Pareto Front of Multilingual Neural Machine Translation](https://mlsanity.com/paper/2304.03216/) (2023)
- [Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets](https://mlsanity.com/paper/2410.04579/) (2024)
- [Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learning](https://mlsanity.com/paper/2210.10293/) (2022)
- [DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization](https://mlsanity.com/paper/2411.14055/) (2024)
- [Distributionally Robust Classification for Multi-source Unsupervised Domain Adaptation](https://mlsanity.com/paper/2601.21315/) (2026)
- [Unifying the Convergences in Multilingual Neural Machine Translation](https://mlsanity.com/paper/2205.01620/) (2022)
- [Out-of-Distribution Generalization in Text Classification: Past, Present, and Future](https://mlsanity.com/paper/2305.14104/) (2023)
- [Is Robustness Transferable across Languages in Multilingual Neural Machine Translation?](https://mlsanity.com/paper/2310.20162/) (2023)

## Abstract (arXiv)

Multilingual neural machine translation (MNMT) learns to translate multiple language pairs with a single model, potentially improving both the accuracy and the memory-efficiency of deployed models. However, the heavy data imbalance between languages hinders the model from performing uniformly across language pairs. In this paper, we propose a new learning objective for MNMT based on distributionally robust optimization, which minimizes the worst-case expected loss over the set of language pairs. We further show how to practically optimize this objective for large translation corpora using an iterated best response scheme, which is both effective and incurs negligible additional computational cost compared to standard empirical risk minimization. We perform extensive experiments on three sets of languages from two datasets and show that our method consistently outperforms strong baseline methods in terms of average and per-language performance under both many-to-one and one-to-many translation settings.