mlsanityData to 7 Oct 2026

V-STaR: Training Verifiers for Self-Taught Reasoners

arXiv 2402.06457 ↗First posted 9 Feb 2024cs.LG cs.AI cs.CLcited by 214

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Common self-improvement approaches for large language models (LLMs), such as STaR, iteratively fine-tune LLMs on self-generated solutions to improve their problem-solving ability. However, these approaches discard the large amounts of incorrect solutions generated during this process, potentially neglecting valuable information in such solutions. To address this shortcoming, we propose V-STaR that utilizes both the correct and incorrect solutions generated during the self-improvement process to train a verifier using DPO that judges correctness of model-generated solutions. This verifier is used at inference time to select one solution among many candidate solutions. Running V-STaR for multiple iterations results in progressively better reasoners and verifiers, delivering a 4% to 17% test accuracy improvement over existing self-improvement and verification approaches on common code generation and math reasoning benchmarks with LLaMA2 models.

Key facts. Cited by 214 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 32 of its 39 references that are in the map. First posted on arXiv 9 Feb 2024.

Builds on

32 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024ReFT: Reasoning with Reinforced Fine-Tuning17 Jan
  2. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  3. 2023Making Large Language Models Better Reasoners with Alignment5 Sep
  4. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2020Learning to summarize from human feedbackcited by 2,562
16 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  4. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  5. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  6. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  7. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  8. 2023Pretraining Language Models with Human Preferencescited by 245
  9. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  10. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  11. 2023Learning From Mistakes Makes LLM Better Reasonercited by 97
  12. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 96
  13. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Modelscited by 76
  14. 2023Improving Large Language Model Fine-tuning for Solving Math Problemscited by 48
  15. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 42
  16. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 58; 2025: 106; 2026: 50.

214 papers.

Sort:

By area:

  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  3. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  4. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  5. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  6. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  7. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  8. 2024Iterative Reasoning Preference Optimizationcited by 244
  9. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  10. 2024Many-Shot In-Context Learningcited by 203
  11. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  12. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
38 more
  1. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  2. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  3. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  4. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  5. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  6. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
  7. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  8. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  9. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  10. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 106
  11. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  12. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  13. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  14. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 96
  15. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  16. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  17. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  18. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoningcited by 85
  19. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  20. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  21. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  22. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  23. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 73
  24. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  25. 2024Vision-Language Models Can Self-Improve Reasoning via Reflectioncited by 66
  26. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  27. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  28. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  29. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  30. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  31. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  32. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  33. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  34. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 44
  35. 2025Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Aheadcited by 43
  36. 2025Dynamic Parallel Tree Search for Efficient LLM Reasoningcited by 42
  37. 2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 41
  38. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40

50 most cited of 214 All 214 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocation
  2. 2025Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection
  3. 2023Teaching Language Models to Self-Improve through Interactive Demonstrations
  4. 2026Verifier-Backed Hard Problem Generation for Mathematical Reasoning
  5. 2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation
  6. 2025OpenSIR: Open-Ended Self-Improving Reasoner
  7. 2022Large Language Models Can Self-Improve
  8. 2024LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback