mlsanityData to 7 Oct 2026

DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining

arXiv 2305.10429 ↗First posted 17 May 2023cs.CL cs.LGcited by 385

Not on the map. Why?

Cited by 24 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The mixture proportions of pretraining data domains (e.g., Wikipedia, books, web text) greatly affect language model (LM) performance. In this paper, we propose Domain Reweighting with Minimax Optimization (DoReMi), which first trains a small proxy model using group distributionally robust optimization (Group DRO) over domains to produce domain weights (mixture proportions) without knowledge of downstream tasks. We then resample a dataset with these domain weights and train a larger, full-sized model. In our experiments, we use DoReMi on a 280M-parameter proxy model to set the domain weights for training an 8B-parameter model (30x larger) more efficiently. On The Pile, DoReMi improves perplexity across all domains, even when it downweights a domain. DoReMi improves average few-shot downstream accuracy by 6.5% points over a baseline model trained using The Pile's default domain weights and reaches the baseline accuracy with 2.6x fewer training steps. On the GLaM dataset, DoReMi, which has no knowledge of downstream tasks, even matches the performance of using domain weights tuned on downstream tasks.

Key facts. Cited by 385 papers in the map (data through 7 Oct 2026). Builds on 42 of its 57 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

42 of 57 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
30 more
  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  2. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  3. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  5. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  7. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  9. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  10. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  11. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  12. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  13. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  14. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  15. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  16. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  17. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  18. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  19. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
  20. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  21. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 224
  22. 2019Distributionally Robust Language Modelingcited by 142
  23. 2022Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?cited by 89
  24. 2021RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learningcited by 72
  25. 2017Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American Englishcited by 69
  26. 2019Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Visioncited by 64
  27. 2019Optimizing Data Usage via Differentiable Rewardscited by 48
  28. 2021Unraveling the Hidden Environmental Impacts of AI Solutions for Environmentcited by 32
  29. 2017Cynical Selection of Language Model Training Datacited by 13
  30. 2022Automatic Document Selection for Efficient Encoder Pretrainingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 34; 2024: 94; 2025: 113; 2026: 144.

385 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  3. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  4. 2023Llemma: An Open Language Model For Mathematicscited by 385
  5. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  6. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  7. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  8. 2023Challenges and Applications of Large Language Modelscited by 251
  9. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  10. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  11. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  12. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
38 more
  1. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  2. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  3. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  4. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  5. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  6. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  7. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  8. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  9. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  10. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 103
  11. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  12. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  13. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  14. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 87
  15. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  16. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  17. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  18. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  19. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  20. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  21. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  22. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 62
  23. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 57
  24. 2023Data-Juicer: A One-Stop Data Processing System for Large Language Modelscited by 52
  25. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 51
  26. 2025Scaling Laws for Optimal Data Mixturescited by 49
  27. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  28. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  29. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 44
  30. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  31. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  32. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  33. 2024Data Selection via Optimal Control for Language Modelscited by 38
  34. 2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37
  35. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  36. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 35
  37. 2024Foundation Models for Music: A Surveycited by 34
  38. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 33

50 most cited of 385 All 385 (JSON)