How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws
On the map: Mixture-of-experts → MoE large language models
Abstract (arXiv)
High-quality data is scarce in large language model (LLM) training, yet how to schedule its use with optimization dynamics lacks theoretical guidance. We extend functional scaling laws with time-varying data quality and derive asymptotically optimal joint data-quality and batch-size schedules within a feature-space regression model. The solution reveals two regimes and dual uses of high-quality data: in the noise-limited regime, a smaller batch converts cleaner data into more signal at comparable noise; in the signal-limited regime, late placement suppresses terminal noise without sacrificing signal accumulation. This explains why conventional decay schedules can conflict with curriculum-style pipelines. Motivated by the theoretical structure, we propose Drop-Stable-Rampup for LLM midtraining: drop the batch size at the quality transition, keep it low to accumulate signal, then ramp up to suppress noise. On a 15B MoE model midtrained on 108B tokens of general-domain proprietary data, Drop-Stable-Rampup improves average accuracy over Warmup-Stable-Decay by +1.70 and Cosine-decay by +2.98, including +4.23 on GSM8K and +2.80 on MATH. On a public math-and-code mixture, it leads all reported STEM, mathematics, and code benchmarks, improving the overall mean over the strongest baseline by +3.27 on a 600M dense model and +5.25 on the same MoE architecture.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 27 of its 38 references that are in the map. First posted on arXiv 25 May 2026.
Builds on
27 of 38 references.
References · 27
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
15 more
- 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
- 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
- 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
- 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
- 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
- 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
- 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
- 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 34
- 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
- 2021Power-law escape rate of SGDcited by 29
- 2018Continuous-time Models for Stochastic Optimization Algorithmscited by 16
- 2025Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedulescited by 14
- 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 12
- 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024How to Train Data-Efficient LLMs
- 2025LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
- 2025LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning
- 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis
- 2026Efficient Scaling of LLM Training with Flexible Context Parallelism
- 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
- 2024Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs
- 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls