mlsanityData to 7 Oct 2026

Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling

arXiv 2408.16737 ↗First posted 29 Aug 2024cs.CL cs.AIcited by 60

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Training on high-quality synthetic data from strong language models (LMs) is a common strategy to improve the reasoning performance of LMs. In this work, we revisit whether this strategy is compute-optimal under a fixed inference budget (e.g., FLOPs). To do so, we investigate the trade-offs between generating synthetic data using a stronger but more expensive (SE) model versus a weaker but cheaper (WC) model. We evaluate the generated data across three key metrics: coverage, diversity, and false positive rate, and show that the data from WC models may have higher coverage and diversity, but also exhibit higher false positive rates. We then finetune LMs on data from SE and WC models in different settings: knowledge distillation, self-improvement, and a novel weak-to-strong improvement setup where a weaker LM teaches reasoning to a stronger LM. Our findings reveal that models finetuned on WC-generated data consistently outperform those trained on SE-generated data across multiple benchmarks and multiple choices of WC and SE models. These results challenge the prevailing practice of relying on SE models for synthetic data generation, suggesting that WC may be the compute-optimal approach for training advanced LM reasoners.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 50 of its 62 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

50 of 62 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking14 Mar
  2. 2023Code Llama: Open Foundation Models for Code24 Aug
  3. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Language20 Dec

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2023Let's Verify Step by Stepcited by 3,598
35 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2024Qwen2 Technical Reportcited by 2,076
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  6. 2024Mixtral of Expertscited by 1,785
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  12. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  13. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  14. 2024Yi: Open Foundation Models by 01.AIcited by 706
  15. 2022Large Language Models Can Self-Improvecited by 656
  16. 2024Self-Rewarding Language Modelscited by 654
  17. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  18. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  19. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  20. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  21. 2023Llemma: An Open Language Model For Mathematicscited by 385
  22. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  23. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  24. 2024Iterative Reasoning Preference Optimizationcited by 244
  25. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  26. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  27. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  28. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 157
  29. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  30. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  31. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  32. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  33. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 41
  34. 2024Weak-to-Strong Reasoningcited by 32
  35. 2024ReMI: A Dataset for Reasoning with Multiple Imagescited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 32; 2026: 8.

60 papers.

Sort:

By area:

  1. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  2. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  3. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
  4. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  5. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  6. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  7. 2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 41
  8. 2025Sleep-time Compute: Beyond Inference Scaling at Test-timecited by 41
  9. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
  10. 2024A Simple Model of Inference Scaling Lawscited by 36
  11. 2025Dynamic Scaling of Unit Tests for Code Reward Modelingcited by 31
  12. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
38 more
  1. 2024Not All LLM Reasoners Are Created Equalcited by 29
  2. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 28
  3. 2025SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugscited by 22
  4. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoningcited by 19
  5. 2024Quantifying Generalization Complexity for Large Language Modelscited by 18
  6. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 16
  7. 2025Superficial Self-Improved Reasoners Benefit from Model Mergingcited by 16
  8. 2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMscited by 14
  9. 2024Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Samplingcited by 13
  10. 2025Weak-to-Strong Generalization Even in Random Feature Networks, Provablycited by 13
  11. 2025HoneyBee: Data Recipes for Vision-Language Reasonerscited by 13
  12. 2024Little Giants: Synthesizing High-Quality Embedding Data at Scalecited by 11
  13. 2025Syntriever: How to Train Your Retriever with Synthetic Data from LLMscited by 11
  14. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  15. 2024LLMs for Cold-Start Cutting Plane Separator Configurationcited by 8
  16. 2024Multi-Agent Sampling: Scaling Inference Compute for Data Synthesis with Tree Search-Based Agentic Collaborationcited by 8
  17. 2024Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Explorationcited by 7
  18. 2025Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based Perspectivecited by 7
  19. 2025DiMA: An LLM-Powered Ride-Hailing Assistant at DiDicited by 7
  20. 2025Enhancing Reasoning Abilities of Small LLMs with Cognitive Alignmentcited by 7
  21. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 7
  22. 2025Knowledge Augmented Complex Problem Solving with Large Language Models: A Surveycited by 6
  23. 2026Squeeze Evolve: Unified Multi-Model Orchestration for Verifier-Free Evolutioncited by 6
  24. 2024LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Modelscited by 4
  25. 2025Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solvingcited by 4
  26. 2025EnsemW2S: Enhancing Weak-to-Strong Generalization with Large Language Model Ensemblescited by 4
  27. 2025AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMscited by 4
  28. 2025TsLLM: Augmenting LLMs for General Time Series Understanding and Predictioncited by 4
  29. 2024Differentially Private Learning Needs Better Model Initialization and Self-Distillationcited by 3
  30. 2025MapCoder-Lite: Distilling Multi-Agent Coding into a Single Small LLMcited by 3
  31. 2024Computational Bottlenecks of Training Small-scale Large Language Modelscited by 2
  32. 2024Guiding Through Complexity: What Makes Good Supervision for Hard Math Reasoning Tasks?cited by 2
  33. 2025Test-Time Scaling with Repeated Sampling Improves Multilingual Text Generationcited by 2
  34. 2025Chart-CoCa: Self-Improving Chart Understanding of Vision LMs via Code-Driven Synthesis and Candidate-Conditioned Answeringcited by 2
  35. 2024EnsemW2S: Enhancing Weak-to-Strong Generalization with Large Language Model Ensemblescited by 1
  36. 2024Composite Learning Units: Generalized Learning Beyond Parameter Updates to Transform LLMs into Adaptive Reasonerscited by 1
  37. 2024Adaptable and Precise: Enterprise-Scenario LLM Function-Calling Capability Training Pipelinecited by 1
  38. 2025Training with Pseudo-Code for Instruction Followingcited by 1

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Inference Scaling vs Reasoning: An Empirical Analysis of Compute-Optimal LLM Problem-Solving
  2. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs
  3. 2025Think, Prune, Train, Improve: Scaling Reasoning without Scaling Models
  4. 2026It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
  5. 2025Towards Reasoning Ability of Small Language Models
  6. 2025Less is More: Improving LLM Reasoning with Minimal Test-Time Intervention
  7. 2026ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation
  8. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code