mlsanityData to 7 Oct 2026

ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search

arXiv 2406.03816 ↗First posted 6 Jun 2024cs.CLcited by 338

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Recent methodologies in LLM self-training mostly rely on LLM generating responses and filtering those with correct output answers as training data. This approach often yields a low-quality fine-tuning training set (e.g., incorrect plans or intermediate reasoning). In this paper, we develop a reinforced self-training approach, called ReST-MCTS*, based on integrating process reward guidance with tree search MCTS* for collecting higher-quality reasoning traces as well as per-step value to train policy and reward models. ReST-MCTS* circumvents the per-step manual annotation typically used to train process rewards by tree-search-based reinforcement learning: Given oracle final correct answers, ReST-MCTS* is able to infer the correct process rewards by estimating the probability this step can help lead to the correct answer. These inferred rewards serve dual purposes: they act as value targets for further refining the process reward model and also facilitate the selection of high-quality traces for policy model self-training. We first show that the tree-search policy in ReST-MCTS* achieves higher accuracy compared with prior LLM reasoning baselines such as Best-of-N and Tree-of-Thought, within the same search budget. We then show that by using traces searched by this tree-search policy as training data, we can continuously enhance the three language models for multiple iterations, and outperform other self-training algorithms such as ReSTEM and Self-Rewarding LM. We release all code at https://github.com/THUDM/ReST-MCTS.

Key facts. Cited by 338 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 54 of its 66 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

54 of 66 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing1 Feb
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  3. 2023Let's Verify Step by Step31 May
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  6. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
36 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2023Mistral 7Bcited by 3,065
  3. 2017Decoupled Weight Decay Regularizationcited by 1,722
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  5. 2023LIMA: Less Is More for Alignmentcited by 1,080
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  7. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  10. 2022Galactica: A Large Language Model for Sciencecited by 736
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  12. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  13. 2024Self-Rewarding Language Modelscited by 654
  14. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  15. 2023Textbooks Are All You Needcited by 561
  16. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  17. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  18. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  19. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  20. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  21. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  22. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  23. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  24. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  25. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  26. 2024Iterative Reasoning Preference Optimizationcited by 244
  27. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  28. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  29. 2022Faithful Reasoning Using Large Language Modelscited by 130
  30. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  31. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  32. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 112
  33. 2024SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Modelscited by 38
  34. 2023Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Designcited by 16
  35. 2024AutoRE: Document-Level Relation Extraction with Large Language Modelscited by 16
  36. 2018Innateness, AlphaZero, and Artificial Intelligencecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 38; 2025: 211; 2026: 89.

338 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  2. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  3. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  4. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  5. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  6. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  7. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  8. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  9. 2024Tree Search for Language Model Agentscited by 148
  10. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  11. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  12. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
38 more
  1. 2025Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Designcited by 100
  2. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 94
  3. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  4. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  5. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  6. 2024Process Reward Model with Q-Value Rankingscited by 78
  7. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  8. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  9. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 75
  10. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  11. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  12. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  13. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 65
  14. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  15. 2024Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Samplingcited by 63
  16. 2024Interpretable Contrastive Monte Carlo Tree Search Reasoningcited by 62
  17. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 62
  18. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  19. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  20. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  21. 2024Synthetic continued pretrainingcited by 56
  22. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  23. 2025DataSciBench: An LLM Agent Benchmark for Data Sciencecited by 53
  24. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  25. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
  26. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  27. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  28. 2025Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Searchcited by 48
  29. 2025A Survey of Large Language Model Agents for Question Answeringcited by 46
  30. 2026AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimizationcited by 46
  31. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 44
  32. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  33. 2025Dynamic Parallel Tree Search for Efficient LLM Reasoningcited by 42
  34. 2024CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repaircited by 41
  35. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  36. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  37. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 39
  38. 2024Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoningcited by 38

50 most cited of 338 All 338 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search
  2. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
  3. 2026ExpRL: Exploratory RL for LLM Mid-Training
  4. 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning
  5. 2026Verified Critical Step Optimization for LLM Agents
  6. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
  7. 2026Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
  8. 2025Unifying Tree Search Algorithm and Reward Design for LLM Reasoning: A Survey