mlsanityData to 7 Oct 2026

How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws

arXiv 2605.25698 ↗First posted 25 May 2026cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

High-quality data is scarce in large language model (LLM) training, yet how to schedule its use with optimization dynamics lacks theoretical guidance. We extend functional scaling laws with time-varying data quality and derive asymptotically optimal joint data-quality and batch-size schedules within a feature-space regression model. The solution reveals two regimes and dual uses of high-quality data: in the noise-limited regime, a smaller batch converts cleaner data into more signal at comparable noise; in the signal-limited regime, late placement suppresses terminal noise without sacrificing signal accumulation. This explains why conventional decay schedules can conflict with curriculum-style pipelines. Motivated by the theoretical structure, we propose Drop-Stable-Rampup for LLM midtraining: drop the batch size at the quality transition, keep it low to accumulate signal, then ramp up to suppress noise. On a 15B MoE model midtrained on 108B tokens of general-domain proprietary data, Drop-Stable-Rampup improves average accuracy over Warmup-Stable-Decay by +1.70 and Cosine-decay by +2.98, including +4.23 on GSM8K and +2.80 on MATH. On a public math-and-code mixture, it leads all reported STEM, mathematics, and code benchmarks, improving the overall mean over the strongest baseline by +3.27 on a 600M dense model and +5.25 on the same MoE architecture.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 27 of its 38 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

27 of 38 references.

References · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  11. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
15 more
  1. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  2. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  3. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  4. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  5. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  6. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  7. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  8. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  9. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 34
  10. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
  11. 2021Power-law escape rate of SGDcited by 29
  12. 2018Continuous-time Models for Stochastic Optimization Algorithmscited by 16
  13. 2025Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedulescited by 14
  14. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 12
  15. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024How to Train Data-Efficient LLMs
  2. 2025LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
  3. 2025LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning
  4. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis
  5. 2026Efficient Scaling of LLM Training with Flexible Context Parallelism
  6. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
  7. 2024Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs
  8. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls