mlsanityData to 7 Oct 2026

UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining

arXiv 2304.09151 ↗First posted 18 Apr 2023cs.CLcited by 147

Not on the map. Why?

Cited by 23 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pretrained multilingual large language models have typically used heuristic temperature-based sampling to balance between different languages. However previous work has not systematically evaluated the efficacy of different pretraining language distributions across model scales. In this paper, we propose a new sampling method, UniMax, that delivers more uniform coverage of head languages while mitigating overfitting on tail languages by explicitly capping the number of repeats over each language's corpus. We perform an extensive series of ablations testing a range of sampling strategies on a suite of multilingual benchmarks, while varying model scale. We find that UniMax outperforms standard temperature-based sampling, and the benefits persist as scale increases. As part of our contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UniMax sampling.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Builds on 43 of its 47 references that are in the map. First posted on arXiv 18 Apr 2023.

Builds on

43 of 47 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
31 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  2. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  3. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  4. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  6. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  8. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  9. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  10. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  11. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  12. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  13. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  14. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  15. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 167
  16. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  17. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  18. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  19. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  20. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  21. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  22. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  23. 2020Balancing Training for Multilingual Neural Machine Translationcited by 71
  24. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  25. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  26. 2019Optimizing Data Usage via Differentiable Rewardscited by 48
  27. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  28. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  29. 2022A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bankcited by 29
  30. 2019Adaptive Scheduling for Multi-Task Learningcited by 24
  31. 2021Balancing Average and Worst-case Accuracy in Multitask Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 23; 2025: 51; 2026: 63.

147 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  2. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  3. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  4. 2025SkyReels-V2: Infinite-length Film Generative Modelcited by 218
  5. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  6. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  7. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  8. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 100
  9. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 89
  10. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 75
  11. 2025Yume: An Interactive World Generation Modelcited by 73
  12. 2025LongCat-Video Technical Reportcited by 73
38 more
  1. 2025Large Video Planner Enables Generalizable Robot Controlcited by 67
  2. 2025WorldGym: World Model as An Environment for Policy Evaluationcited by 61
  3. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 58
  4. 2025ACE-Step: A Step Towards Music Generation Foundation Modelcited by 54
  5. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
  6. 2025Scaling Laws for Optimal Data Mixturescited by 49
  7. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  8. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  9. 2025HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioningcited by 44
  10. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  11. 2026NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videoscited by 39
  12. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  13. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
  14. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 35
  15. 2025FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Predictioncited by 33
  16. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
  17. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  18. 2024MERA: A Comprehensive LLM Evaluation in Russiancited by 22
  19. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 22
  20. 2026ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytellingcited by 22
  21. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 21
  22. 2024Uni-Mol2: Exploring Molecular Pretraining Model at Scalecited by 20
  23. 2025Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Modelscited by 19
  24. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 19
  25. 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
  26. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  27. 2025Yan: Foundational Interactive Video Generationcited by 15
  28. 2023PersianLLaMA: Towards Building First Persian Large Language Modelcited by 14
  29. 2024YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentationcited by 13
  30. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  31. 2026ActWorld: From Explorable to Interactive World Model via Action-Aware Memorycited by 13
  32. 2026Wonder: Video World Model Done Bettercited by 13
  33. 2025MagicTryOn: Harnessing Diffusion Transformer for Garment-Preserving Video Virtual Try-oncited by 12
  34. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  35. 2024SambaLingo: Teaching Large Language Models New Languagescited by 11
  36. 2024Data, Data Everywhere: A Guide for Pretraining Dataset Constructioncited by 11
  37. 2023mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequencescited by 10
  38. 2025Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindicited by 10

50 most cited of 147 All 147 (JSON)