# Balancing Training for Multilingual Neural Machine Translation

arXiv: https://arxiv.org/abs/2004.06748 · HTML: https://mlsanity.com/paper/2004.06748/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2020.

## Built on by (most cited first, 30 of 72)

- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity](https://mlsanity.com/paper/2305.13169/) (2023)
- [UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining](https://mlsanity.com/paper/2304.09151/) (2023)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [What is Wrong with Perplexity for Long-context Language Modeling?](https://mlsanity.com/paper/2410.23771/) (2024)
- [Scaling End-to-End Models for Large-Scale Multilingual ASR](https://mlsanity.com/paper/2104.14830/) (2021)
- [Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs](https://mlsanity.com/paper/2206.04674/) (2022)
- [Multi-Objective Meta Learning](https://mlsanity.com/paper/2102.07121/) (2021)
- [Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills](https://mlsanity.com/paper/2107.07261/) (2021)
- [Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task](https://mlsanity.com/paper/2111.02086/) (2021)
- [Measuring The Impact Of Programming Language Distribution](https://mlsanity.com/paper/2302.01973/) (2023)
- [Many-to-English Machine Translation Tools, Data, and Pretrained Models](https://mlsanity.com/paper/2104.00290/) (2021)
- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [Scaling Laws for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2302.09650/) (2023)
- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [Causes and Cures for Interference in Multilingual Translation](https://mlsanity.com/paper/2212.07530/) (2022)
- [Uncertainty-Aware Balancing for Multilingual and Multi-Domain Neural Machine Translation Training](https://mlsanity.com/paper/2109.02284/) (2021)
- [Distributionally Robust Multilingual Machine Translation](https://mlsanity.com/paper/2109.04020/) (2021)
- [RECKONING: Reasoning through Dynamic Knowledge Encoding](https://mlsanity.com/paper/2305.06349/) (2023)
- [Tricks for Training Sparse Translation Models](https://mlsanity.com/paper/2110.08246/) (2021)
- [Meta Back-translation](https://mlsanity.com/paper/2102.07847/) (2021)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning](https://mlsanity.com/paper/2110.02600/) (2021)
- [Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits](https://mlsanity.com/paper/2110.06997/) (2021)
- [Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models](https://mlsanity.com/paper/2406.08811/) (2024)
- [Robust Optimization for Multilingual Translation with Imbalanced Data](https://mlsanity.com/paper/2104.07639/) (2021)
- [Meta-Learning for Effective Multi-task and Multilingual Modelling](https://mlsanity.com/paper/2101.10368/) (2021)
- [Tencent's Multilingual Machine Translation System for WMT22 Large-Scale African Languages](https://mlsanity.com/paper/2210.09644/) (2022)
- [Learning Language-Specific Layers for Multilingual Machine Translation](https://mlsanity.com/paper/2305.02665/) (2023)
- [Efficient Weight factorization for Multilingual Speech Recognition](https://mlsanity.com/paper/2105.03010/) (2021)

## Abstract (arXiv)

When training multilingual machine translation (MT) models that can translate to/from multiple languages, we are faced with imbalanced training sets: some languages have much more training data than others. Standard practice is to up-sample less resourced languages to increase representation, and the degree of up-sampling has a large effect on the overall performance. In this paper, we propose a method that instead automatically learns how to weight training data through a data scorer that is optimized to maximize performance on all test languages. Experiments on two sets of languages under both one-to-many and many-to-one MT settings show our method not only consistently outperforms heuristic baselines in terms of average performance, but also offers flexible control over the performance of which languages are optimized.