mlsanityData to 7 Oct 2026

Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models

arXiv 2312.06585 ↗First posted 11 Dec 2023cs.LGcited by 325

On the map: Synthetic data and self-improvement for LLMs → Synthetic reasoning data and bootstrapping

Abstract (arXiv)

Fine-tuning language models~(LMs) on human-generated data remains a prevalent practice. However, the performance of such models is often limited by the quantity and diversity of high-quality human data. In this paper, we explore whether we can go beyond human data on tasks where we have access to scalar feedback, for example, on math problems where one can verify correctness. To do so, we investigate a simple self-training method based on expectation-maximization, which we call ReSTEM, where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times. Testing on advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find that ReSTEM scales favorably with model size and significantly surpasses fine-tuning only on human data. Overall, our findings suggest self-training with feedback can substantially reduce dependence on human-generated data.

Key facts. Cited by 325 papers in the map (data through 7 Oct 2026). Placed in Synthetic reasoning data and bootstrapping, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 22 of its 30 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

22 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Large Language Models Can Self-Improve20 Oct
  2. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 20

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2021Measuring Coding Challenge Competence With APPScited by 879
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  11. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  12. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
8 more
  1. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  2. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  3. 2016Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervisioncited by 101
  4. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80
  5. 2023Training Chain-of-Thought via Latent-Variable Inferencecited by 46
  6. 2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
  7. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  8. 2023Joint Prompt Optimization of Stacked LLMs using Variational Inferencecited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 102; 2025: 124; 2026: 99.

325 papers.

Sort:

By area:

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  3. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  4. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  5. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  6. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  7. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  8. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  9. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  10. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  11. 2024LoRA Learns Less and Forgets Lesscited by 313
  12. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
38 more
  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  3. 2024Iterative Reasoning Preference Optimizationcited by 244
  4. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  5. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  6. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  7. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  8. 2024Many-Shot In-Context Learningcited by 203
  9. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  10. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  11. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  12. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  13. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  14. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  15. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
  16. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  17. 2025Process Reward Models That Thinkcited by 118
  18. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 117
  19. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  20. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  21. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  22. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  23. 2024Approaching Human-Level Forecasting with Language Modelscited by 102
  24. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  25. 2025Maximizing Confidence Alone Improves Reasoningcited by 97
  26. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  27. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  28. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  29. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  30. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  31. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  32. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  33. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 81
  34. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  35. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  36. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  37. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMscited by 72
  38. 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 71

50 most cited of 325 All 325 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-training Language Models for Arithmetic Reasoning
  2. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline
  3. 2025A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
  4. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards
  5. 2023Teaching Language Models to Self-Improve through Interactive Demonstrations
  6. 2026Peer-Predictive Self-Training for Language Model Reasoning
  7. 2023Self-Refine: Iterative Refinement with Self-Feedback
  8. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision