mlsanityData to 14 Oct 2026

Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives

arXiv 2505.21598 ↗First posted 27 May 2025cs.CLcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Training large language models with data collected from various domains can improve their performance on downstream tasks. However, given a fixed training budget, the sampling proportions of these different domains significantly impact the model's performance. How can we determine the domain weights across different data domains to train the best-performing model within constrained computational resources? In this paper, we provide a comprehensive overview of existing data mixture methods. First, we propose a fine-grained categorization of existing methods, extending beyond the previous offline and online classification. Offline methods are further grouped into heuristic-based, algorithm-based, and function fitting-based methods. For online methods, we categorize them into three groups: online min-max optimization, online mixing law, and other approaches by drawing connections with the optimization frameworks underlying offline methods. Second, we summarize the problem formulations, representative algorithms for each subtype of offline and online methods, and clarify the relationships and distinctions among them. Finally, we discuss the advantages and disadvantages of each method and highlight key challenges in the field of data mixture.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 36 of its 51 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

36 of 51 references.

References · 36

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  6. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  8. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  9. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  10. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  11. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  12. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 270
24 more
  1. 2024A Survey on Data Selection for Language Modelscited by 236
  2. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  3. 2017Training GANs with Optimismcited by 217
  4. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
  5. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  6. 2019Distributionally Robust Language Modelingcited by 143
  7. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  8. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  9. 2018The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimizationcited by 101
  10. 2023A Fully First-Order Method for Stochastic Bilevel Optimizationcited by 100
  11. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 90
  12. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  13. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 75
  14. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  15. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 60
  16. 2020A Single-Loop Smoothed Gradient Descent-Ascent Algorithm for Nonconvex-Concave Min-Max Problemscited by 58
  17. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 47
  18. 2020Non-convex Min-Max Optimization: Applications, Challenges, and Recent Theoretical Advancescited by 38
  19. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 37
  20. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 34
  21. 2014A Unified Perspective on Multi-Domain and Multi-Task Learningcited by 33
  22. 2024AutoScale: Scale-Aware Data Mixing for Pre-Training LLMscited by 22
  23. 2024The Vizier Gaussian Process Bandit Algorithmcited by 14
  24. 2024Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-trainingcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluationcited by 0
  2. 2026Harnessing the Potential of Optimizing Data Mixtures via Bayesian Domain Reweightingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization
  2. 2024Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models
  3. 2026Rethinking Data Mixing from the Perspective of Large Language Models
  4. 2025Do Large Language Models (Really) Need Statistical Foundations?
  5. 2024Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations
  6. 2025Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging
  7. 2024A Survey on Mixture of Experts in Large Language Models
  8. 2026Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time