mlsanityData to 7 Oct 2026

Efficient Online Data Mixing For Language Model Pre-Training

arXiv 2312.02406 ↗First posted 5 Dec 2023cs.CL cs.LGcited by 77

Not on the map. Why?

Abstract (arXiv)

The data used to pretrain large language models has a decisive impact on a model's downstream performance, which has led to a large body of work on data selection methods that aim to automatically determine the most suitable data to use for pretraining. Existing data selection methods suffer from slow and computationally expensive processes, a problem amplified by the increasing size of models and of pretraining datasets. Data mixing, on the other hand, reduces the complexity of data selection by grouping data points together and determining sampling probabilities across entire groups. However, data mixing proportions are typically fixed before training and therefore cannot adapt to changing training dynamics. To address these limitations, we develop an efficient algorithm for Online Data Mixing (ODM) that combines elements from both data selection and data mixing. Based on multi-armed bandit algorithms, our online approach optimizes the data mixing proportions during training. Remarkably, our method trains a model that reaches the final perplexity of the next best method with 19% fewer training iterations, and improves performance on the 5-shot MMLU benchmark by 1.9% relative accuracy, while adding negligible wall-clock time during pretraining.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

12 of 17 references.

References · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  9. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  10. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  11. 2021A Few More Examples May Be Worth Billions of Parameterscited by 19
  12. 2023Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Datacited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 37; 2026: 20.

77 papers.

Sort:

By area:

  1. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  2. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  3. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  4. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 104
  5. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 103
  6. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  7. 2025Scaling Laws for Optimal Data Mixturescited by 49
  8. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  9. 2024Data Selection via Optimal Control for Language Modelscited by 38
  10. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
  11. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 33
  12. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
38 more
  1. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  2. 2024AutoScale: Scale-Aware Data Mixing for Pre-Training LLMscited by 20
  3. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Expertscited by 19
  4. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
  5. 2025Citrus: Leveraging Expert Cognitive Pathways in a Medical Language Model for Advanced Medical Decision Supportcited by 16
  6. 2025Efficient Data Selection at Scale via Influence Distillationcited by 14
  7. 2025Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuningcited by 14
  8. 2025Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Datacited by 13
  9. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  10. 2025Data Mixing Optimization for Supervised Fine-Tuning of Large Language Modelscited by 12
  11. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  12. 2024Data, Data Everywhere: A Guide for Pretraining Dataset Constructioncited by 11
  13. 2024Scalable Data Ablation Approximations for Language Models through Modular Training and Mergingcited by 11
  14. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 11
  15. 2025MixMin: Finding Data Mixtures via Convex Minimizationcited by 10
  16. 2025Revisiting Multilingual Data Mixtures in Language Model Pretrainingcited by 10
  17. 2025Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Frameworkcited by 8
  18. 2025AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMscited by 8
  19. 2024Towards Optimal Learning of Language Modelscited by 7
  20. 2025Merge to Mix: Mixing Datasets via Model Mergingcited by 7
  21. 2025Fast Data Attribution for Text-to-Image Modelscited by 7
  22. 2025DUET: Optimizing Training Data Mixtures via Feedback from Unseen Evaluation Taskscited by 6
  23. 2025DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collectionscited by 6
  24. 2025TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-trainingcited by 6
  25. 2025Train Once, Answer All: Many Pretraining Experiments for the Cost of Onecited by 6
  26. 2025Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Modelscited by 5
  27. 2025Shift is Good: Mismatched Data Mixing Improves Test Performancecited by 5
  28. 2024DiPT: Enhancing LLM reasoning through diversified perspective-takingcited by 4
  29. 2024Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasetscited by 4
  30. 2024Measuring Fingerprints of Web-filtered Text Datasets and Fingerprint Propagation Through Trainingcited by 4
  31. 2025GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretrainingcited by 4
  32. 2026DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Modelscited by 4
  33. 2024DEM: Distribution Edited Model for Training with Mixed Data Distributionscited by 3
  34. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  35. 2025ToReMi: Topic-Aware Data Reweighting for Dynamic Pre-Training Data Selectioncited by 3
  36. 2025MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sourcescited by 3
  37. 2025BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretrainingcited by 3
  38. 2025ADAPT: Learning Task Mixtures for Budget-Constrained Instruction Tuningcited by 3

50 most cited of 77 All 77 (JSON)