mlsanityData to 7 Oct 2026

RegMix: Data Mixture as Regression for Language Model Pre-training

arXiv 2407.01492 ↗First posted 1 Jul 2024cs.CL cs.AIcited by 172

Not on the map. Why?

Abstract (arXiv)

The data mixture for large language model pre-training significantly impacts performance, yet how to determine an effective mixture remains unclear. We propose RegMix to automatically identify a high-performing data mixture by formulating it as a regression task. RegMix trains many small models on diverse data mixtures, uses regression to predict performance of unseen mixtures, and applies the best predicted mixture to train a large-scale model with orders of magnitude more compute. To empirically validate RegMix, we train 512 models with 1M parameters for 1B tokens to fit the regression model and predict the best data mixture. Using this mixture we train a 1B parameter model for 25B tokens (i.e. 1000x larger and 25x longer) which we find performs best among 64 candidate 1B parameter models with other mixtures. Furthermore, RegMix consistently outperforms human selection in experiments involving models up to 7B models trained on 100B tokens, while matching or exceeding DoReMi using just 10% of the computational resources. Our experiments also show that (1) Data mixtures significantly impact performance; (2) Web corpora rather than data perceived as high-quality like Wikipedia have the strongest positive correlation with downstream performance; (3) Domains interact in complex ways often contradicting common sense, thus automatic approaches like RegMix are needed; (4) Data mixture effects transcend scaling laws. Our code is available at https://github.com/sail-sg/regmix.

Key facts. Cited by 172 papers in the map (data through 7 Oct 2026). Builds on 60 of its 71 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

60 of 71 references.

References · 60

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  4. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  7. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  8. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  9. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  11. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  12. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  13. 2023Scaling Data-Constrained Language Modelscited by 391
  14. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  15. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  16. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  17. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  18. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  19. 2024A Survey on Data Selection for Language Modelscited by 230
  20. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  21. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  22. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  23. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  24. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  25. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  26. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 103
  27. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  28. 2024How to Train Data-Efficient LLMscited by 91
  29. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 87
  30. 2022Training Trajectories of Language Models Across Scalescited by 80
  31. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  32. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  33. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 74
  34. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
  35. 2024Compression Represents Intelligence Linearlycited by 56
  36. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
  37. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  38. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 75; 2026: 82.

172 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  3. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  4. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 120
  5. 2024Scaling Laws for Precisioncited by 92
  6. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  7. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 62
  8. 2025Scaling Laws for Optimal Data Mixturescited by 49
  9. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  10. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  11. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  12. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
38 more
  1. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 33
  2. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
  3. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
  4. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 24
  5. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 22
  6. 2024AutoScale: Scale-Aware Data Mixing for Pre-Training LLMscited by 20
  7. 2025Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Modelscited by 19
  8. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  9. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 19
  10. 2024A Survey on Mixup Augmentations and Beyondcited by 18
  11. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
  12. 2025A Survey on LLM Mid-Trainingcited by 18
  13. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  14. 2025Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correctioncited by 16
  15. 2025QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretrainingcited by 16
  16. 2025Bag of Tricks for Inference-time Computation of LLM Reasoningcited by 15
  17. 2025Data Mixing Can Induce Phase Transitions in Knowledge Acquisitioncited by 15
  18. 2025Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuningcited by 14
  19. 2025Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Datacited by 13
  20. 2025Q-SiT: Teaching LMMs for Image Quality Scoring and Interpretingcited by 13
  21. 2025Group-Level Data Selection for Efficient Pretrainingcited by 12
  22. 2025Data Mixing Optimization for Supervised Fine-Tuning of Large Language Modelscited by 12
  23. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 12
  24. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 11
  25. 2024Dynamic Gradient Alignment for Online Data Mixingcited by 10
  26. 2024Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaborationcited by 10
  27. 2025MixMin: Finding Data Mixtures via Convex Minimizationcited by 10
  28. 2025SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversitycited by 10
  29. 2025Guiding Data Collection via Factored Scaling Curvescited by 10
  30. 2025Revisiting Multilingual Data Mixtures in Language Model Pretrainingcited by 10
  31. 2025OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoningcited by 10
  32. 2026GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learningcited by 10
  33. 2025PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflictscited by 8
  34. 2025Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Frameworkcited by 8
  35. 2025ZClip: Adaptive Spike Mitigation for LLM Pre-Trainingcited by 8
  36. 2025IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignmentcited by 8
  37. 2025AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMscited by 8
  38. 2025Merge to Mix: Mixing Datasets via Model Mergingcited by 7

50 most cited of 172 All 172 (JSON)