mlsanityData to 7 Oct 2026

Compute Optimal Scaling of Skills: Knowledge vs Reasoning

arXiv 2503.10061 ↗First posted 13 Mar 2025cs.LG cs.AI cs.CLcited by 11

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Scaling laws are a critical component of the LLM development pipeline, most famously as a way to forecast training decisions such as 'compute-optimally' trading-off parameter count and dataset size, alongside a more recent growing list of other crucial decisions. In this work, we ask whether compute-optimal scaling behaviour can be skill-dependent. In particular, we examine knowledge and reasoning-based skills such as knowledge-based QA and code generation, and we answer this question in the affirmative: scaling laws are skill-dependent. Next, to understand whether skill-dependent scaling is an artefact of the pretraining datamix, we conduct an extensive ablation of different datamixes and find that, also when correcting for datamix differences, knowledge and code exhibit fundamental differences in scaling behaviour. We conclude with an analysis of how our findings relate to standard compute-optimal scaling using a validation set, and find that a misspecified validation set can impact compute-optimal parameter count by nearly 50%, depending on its skill composition.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 35 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

27 of 35 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  11. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  12. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
15 more
  1. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  2. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  3. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  4. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  5. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  6. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  7. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  8. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  9. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  10. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  11. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 33
  12. 2024Establishing Task Scaling Laws via Compute-Efficient Model Ladderscited by 32
  13. 2023CAT-LM: Training Language Models on Aligned Code And Testscited by 28
  14. 2024Sloth: scaling laws for LLM skills to predict multi-benchmark performance across familiescited by 23
  15. 2025Optimizing Pretraining Data Mixtures with LLM-Estimated Utilitycited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 22
  2. 2025Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Modelscited by 3
  3. 2025R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Trainingcited by 3
  4. 2025Scaling Laws for Code: A More Data-Hungry Regimecited by 3
  5. 2026The Scaling Laws of Skills in LLM Agent Systemscited by 3
  6. 2025A Latent Variable Framework for Scaling Laws in Large Language Modelscited by 2
  7. 2025LLM-Integrated Bayesian State Space Models for Multimodal Time-Series Forecastingcited by 1
  8. 2026Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimizationcited by 1
  9. 2025Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Taskscited by 0
  10. 2025Transformers Can Learn Connectivity in Some Graphs but Not Otherscited by 0
  11. 2026BatteryPass-12K: The First Dataset for the Novel Digital Battery Passport Conformance Taskcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
  2. 2026Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning
  3. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search
  4. 2025Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning
  5. 2025How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients
  6. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition
  7. 2026Test-Time Scaling Makes Overtraining Compute-Optimal
  8. 2026Prescriptive Scaling Reveals the Evolution of Language Model Capabilities