mlsanityData to 14 Oct 2026

Order Matters in the Presence of Dataset Imbalance for Multilingual Learning

arXiv 2312.06134 ↗First posted 11 Dec 2023cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

In this paper, we empirically study the optimization dynamics of multi-task learning, particularly focusing on those that govern a collection of tasks with significant data imbalance. We present a simple yet effective method of pre-training on high-resource tasks, followed by fine-tuning on a mixture of high/low-resource tasks. We provide a thorough empirical study and analysis of this method's benefits showing that it achieves consistent improvements relative to the performance trade-off profile of standard static weighting. We analyze under what data regimes this method is applicable and show its improvements empirically in neural machine translation (NMT) and multi-lingual language modeling.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 28 of its 32 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

28 of 32 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2021Convex optimizationcited by 2,515
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  9. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  10. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
  11. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  12. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
16 more
  1. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  2. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
  3. 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
  4. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  5. 2020Balancing Training for Multilingual Neural Machine Translationcited by 72
  6. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  7. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
  8. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
  9. 2018Trivial Transfer Learning for Low-Resource Neural Machine Translationcited by 47
  10. 2020Multi-task Learning for Multilingual Neural Machine Translationcited by 46
  11. 2020English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Toocited by 39
  12. 2019Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabulariescited by 27
  13. 2019Adaptive Scheduling for Multi-Task Learningcited by 24
  14. 2018Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabularycited by 18
  15. 2022Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translationcited by 18
  16. 2021Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Banditscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2025Revisiting Multilingual Data Mixtures in Language Model Pretrainingcited by 10
  2. 2025Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learningcited by 9
  3. 2024Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasetscited by 6
  4. 2025CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignmentcited by 3
  5. 2026"UberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Datasetcited by 3
  6. 2025Using Temperature Sampling to Effectively Train Robot Learning Policies on Imbalanced Datasetscited by 0
  7. 2026Influence Guided Sampling for Domain Adaptation of Text Retrieverscited by 0