mlsanityData to 7 Oct 2026

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance

arXiv 2403.16952 ↗First posted 25 Mar 2024cs.CL cs.AI cs.LGcited by 179

Not on the map. Why?

Abstract (arXiv)

Pretraining data of large language models composes multiple domains (e.g., web texts, academic papers, codes), whose mixture proportions crucially impact the competence of outcome models. While existing endeavors rely on heuristics or qualitative strategies to tune the proportions, we discover the quantitative predictability of model performance regarding the mixture proportions in function forms, which we refer to as the data mixing laws. Fitting such functions on sample mixtures unveils model performance on unseen mixtures before actual runs, thus guiding the selection of an ideal data mixture. Furthermore, we propose nested use of the scaling laws of training steps, model sizes, and our data mixing law to enable predicting the performance of large models trained on massive data under various mixtures with only small-scale training. Moreover, experimental results verify that our method effectively optimizes the training mixture of a 1B model trained for 100B tokens in RedPajama, reaching a performance comparable to the one trained for 48% more steps on the default mixture. Extending the application of data mixing laws to continual training accurately predicts the critical mixture proportion that avoids catastrophic forgetting and outlooks the potential for dynamic data schedules

Key facts. Cited by 179 papers in the map (data through 7 Oct 2026). Builds on 48 of its 74 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

48 of 74 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023Mistral 7Bcited by 3,065
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
36 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  6. 2022Galactica: A Large Language Model for Sciencecited by 736
  7. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  8. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  11. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  12. 2023Scaling Data-Constrained Language Modelscited by 391
  13. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  14. 2021Explaining Neural Scaling Lawscited by 353
  15. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  16. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  17. 2021Scaling Laws for Transfercited by 265
  18. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  19. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  20. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  21. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  22. 2023The Quantization Model of Neural Scalingcited by 137
  23. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  24. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
  25. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  26. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  27. 2022Broken Neural Scaling Lawscited by 102
  28. 2021Learning Curve Theorycited by 79
  29. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  30. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  31. 2023Data-Juicer: A One-Stop Data Processing System for Large Language Modelscited by 52
  32. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  33. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 41
  34. 2024Unraveling the Mystery of Scaling Laws: Part Icited by 24
  35. 2024Balanced Data Sampling for Language Model Training with Clusteringcited by 22
  36. 2024Towards Optimal Learning of Language Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 67; 2026: 82.

179 papers.

Sort:

By area:

  1. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  2. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  3. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  4. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  5. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 59
  6. 2025Scaling Laws for Optimal Data Mixturescited by 49
  7. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  8. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  9. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  10. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 33
  11. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  12. 2024Towards Neural Scaling Laws for Time Series Foundation Modelscited by 30
38 more
  1. 2025Parallel Scaling Law for Language Modelscited by 29
  2. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
  3. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  4. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Modelscited by 25
  5. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 24
  6. 2024CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Modelscited by 23
  7. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  8. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  9. 2024AutoScale: Scale-Aware Data Mixing for Pre-Training LLMscited by 20
  10. 2025E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-trainingcited by 20
  11. 2025Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Modelscited by 19
  12. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  13. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 19
  14. 2025Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correctioncited by 16
  15. 2025QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretrainingcited by 16
  16. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  17. 2025Data Mixing Can Induce Phase Transitions in Knowledge Acquisitioncited by 15
  18. 2025Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuningcited by 14
  19. 2025Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Datacited by 13
  20. 2025Q-SiT: Teaching LMMs for Image Quality Scoring and Interpretingcited by 13
  21. 2025Can Language Models Discover Scaling Laws?cited by 12
  22. 2025Data Mixing Optimization for Supervised Fine-Tuning of Large Language Modelscited by 12
  23. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 12
  24. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  25. 2024Temporal Scaling Law for Large Language Modelscited by 11
  26. 2024Scalable Data Ablation Approximations for Language Models through Modular Training and Mergingcited by 11
  27. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 11
  28. 2025Compute Optimal Scaling of Skills: Knowledge vs Reasoningcited by 11
  29. 2024Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaborationcited by 10
  30. 2025SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversitycited by 10
  31. 2026GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learningcited by 10
  32. 2024AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systemscited by 9
  33. 2025Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessonscited by 9
  34. 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoningcited by 9
  35. 2024Demystifying Language Model Forgetting with Low-rank Example Associationscited by 8
  36. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  37. 2024Are Bigger Encoders Always Better in Vision Large Models?cited by 8
  38. 2025PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflictscited by 8

50 most cited of 179 All 179 (JSON)