mlsanityData to 14 Oct 2026

AC-ODM: Actor--Critic Online Data Mixing for Sample-Efficient LLM Pretraining

arXiv 2505.23878 ↗First posted 29 May 2025cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Optimizing pretraining data composition is pivotal for LLM generalization. While dynamic mixing outperforms static strategies by capturing evolving training dynamics, current methods fail to reconcile computational efficiency with sample efficiency and structural flexibility for diverse pipelines.We introduce Actor--Critic Online Data Mixing (AC-ODM), which approaches data mixing from a reinforcement learning perspective with a parameterized policy that we theoretically prove to act as a dynamic linear surrogate maximizing the constructive interference of gradients. To enhance practical flexibility, AC-ODM supports two operational modes: (i) a proxy mode for fixed, pre-prepared corpora, where a policy learned on a small model is transferred to a larger target; and (ii) a non-proxy mode for direct end-to-end training from scratch without priors. Empirically, AC-ODM significantly outperforms prior methods in convergence speed and downstream accuracy across various architectures. On Pythia-1B, it reaches optimal validation perplexity using up to 66% fewer training steps than competitive baselines, delivering a 27.5% relative improvement in MMLU accuracy and a 2.23 x higher pass@1 on HumanEval, all while incurring a virtually negligible (0.4%) per-step wall-clock increase and only 2% additional memory overhead. Code is available at https://github.com/DANG-ai/AC-ODM.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

19 of 34 references.

References · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  10. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  11. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  12. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
7 more
  1. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  2. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  3. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 66
  4. 2025Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Modelscited by 19
  5. 2025QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretrainingcited by 16
  6. 2025Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Frameworkcited by 9
  7. 2025PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflictscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1
  2. 2026Small Models Scout Bottleneck Order for Large-Model Data Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning
  2. 2026Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
  3. 2026Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders
  4. 2026Dr. Post-Training: A Data Regularization Perspective on LLM Post-Training
  5. 2026DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learning
  6. 2026Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages
  7. 2026RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
  8. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization