mlsanityData to 14 Oct 2026

Robust Optimization for Multilingual Translation with Imbalanced Data

arXiv 2104.07639 ↗First posted 15 Apr 2021cs.CL cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Multilingual models are parameter-efficient and especially effective in improving low-resource languages by leveraging crosslingual transfer. Despite recent advance in massive multilingual translation with ever-growing model and data, how to effectively train multilingual models has not been well understood. In this paper, we show that a common situation in multilingual training, data imbalance among languages, poses optimization tension between high resource and low resource languages where the found multilingual solution is often sub-optimal for low resources. We show that common training method which upsamples low resources can not robustly optimize population loss with risks of either underfitting high resource languages or overfitting low resource ones. Drawing on recent findings on the geometry of loss landscape and its effect on generalization, we propose a principled optimization algorithm, Curvature Aware Task Scaling (CATS), which adaptively rescales gradients from different tasks with a meta objective of guiding multilingual training to low-curvature neighborhoods with uniformly low loss for all languages. We ran experiments on common benchmarks (TED, WMT and OPUS-100) with varying degrees of data imbalance. CATS effectively improved multilingual optimization and as a result demonstrated consistent gains on low resources (+0.8 to +2.2 BLEU) without hurting high resources. In addition, CATS is robust to overparameterization and large batch size training, making it a promising training method for massive multilingual models that truly improve low resource languages.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 55 of its 63 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

55 of 63 references.

References · 55

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2016Layer Normalizationcited by 5,669
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
38 more
  1. 2020On Layer Normalization in the Transformer Architecturecited by 922
  2. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  3. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  4. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  7. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  8. 2019How multilingual is Multilingual BERT?cited by 645
  9. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  10. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  11. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  12. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  13. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  14. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  15. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  16. 2018An Empirical Model of Large-Batch Trainingcited by 281
  17. 2014The Loss Surfaces of Multilayer Networkscited by 280
  18. 2018Learning Word Vectors for 157 Languagescited by 280
  19. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  20. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  21. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
  22. 2019Massively Multilingual Neural Machine Translationcited by 246
  23. 2019Pareto Multi-Task Learningcited by 224
  24. 2019Understanding and Improving Layer Normalizationcited by 217
  25. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  26. 2017Multi-task Self-Supervised Visual Learningcited by 200
  27. 2014Qualitatively characterizing neural network optimization problemscited by 188
  28. 2020Understanding the Role of Training Regimes in Continual Learningcited by 180
  29. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  30. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  31. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  32. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 130
  33. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 126
  34. 2015Multi-task Sequence to Sequence Learningcited by 94
  35. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  36. 2020Balancing Training for Multilingual Neural Machine Translationcited by 72
  37. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 62
  38. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 4; 2023: 3; 2024: 4; 2025: 2.

15 papers.

Sort:

By area:

  1. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 86
  2. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
  3. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  4. 2022Causes and Cures for Interference in Multilingual Translationcited by 23
  5. 2022Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translationcited by 21
  6. 2024Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translationcited by 19
  7. 2021Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learningcited by 16
  8. 2021Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Banditscited by 16
  9. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 15
  10. 2025DataRater: Meta-Learned Dataset Curationcited by 13
  11. 2025Vintix: Action Model via In-Context Reinforcement Learningcited by 10
  12. 2023On the Pareto Front of Multilingual Neural Machine Translationcited by 7
3 more
  1. 2024Improving Robustness to Multiple Spurious Correlations by Multi-Objective Optimizationcited by 6
  2. 2022Refining Low-Resource Unsupervised Translation by Language Disentanglement of Multilingual Modelcited by 2
  3. 2024DMTG: One-Shot Differentiable Multi-Task Groupingcited by 0