mlsanityData to 7 Oct 2026

Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search

arXiv 2501.01478 ↗First posted 2 Jan 2025cs.AI cs.CL cs.LGcited by 14

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large language models (LLMs) have demonstrated their remarkable capacity across a variety of tasks. However, reasoning remains a challenge for LLMs. To improve LLMs' reasoning ability, process supervision has proven to be better than outcome supervision. In this work, we study using Monte Carlo Tree Search (MCTS) to generate process supervision data with LLMs themselves for training them. We sample reasoning steps with an LLM and assign each step a score that captures its "relative correctness," and the LLM is then trained by minimizing weighted log-likelihood of generating the reasoning steps. This generate-then-train process is repeated iteratively until convergence.Our experimental results demonstrate that the proposed methods considerably improve the performance of LLMs on two mathematical reasoning datasets. Furthermore, models trained on one dataset also exhibit improved performance on the other, showing the transferability of the enhanced reasoning ability.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 30 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

24 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  2. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  3. 2024AlphaMath Almost Zero: Process Supervision without Process6 May
  4. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  5. 2023Let's Verify Step by Step31 May
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  8. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  9. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  10. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  11. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  12. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
6 more
  1. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  2. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 156
  3. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  4. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 108
  5. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMscited by 72
  6. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 4.

14 papers.

Sort:

By area:

  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  2. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  3. 2025Learning to Reason Across Parallel Samples for LLM Reasoningcited by 32
  4. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  5. 2025Navigating the Alpha Jungle: An LLM-Powered MCTS Framework for Formulaic Factor Miningcited by 13
  6. 2025Investigating Inference-time Scaling for Chain of Multi-modal Thought: A Preliminary Studycited by 11
  7. 2025A Survey on Parallel Reasoningcited by 11
  8. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  9. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  10. 2025VCU-Bridge: Hierarchical Visual Connotation Understanding via Semantic Bridgingcited by 2
  11. 2026Process Supervision of Confidence Margin for Calibrated LLM Reasoningcited by 1
  12. 2026Efficient Reasoning on the Edgecited by 0
2 more
  1. 2026Deep Interaction: An Efficient Human-AI Interaction Method for Large Reasoning Modelscited by 0
  2. 2026Reflective Recovery: A Self-Supervised Method for Reasoning by Learning from Mistakescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision
  2. 2026Process Supervision for Chain-of-Thought Reasoning via Monte Carlo Net Information Gain
  3. 2025From Static to Dynamic: Adaptive Monte Carlo Search for Mathematical Process Supervision
  4. 2025An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning
  5. 2023No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Function
  6. 2025CMCTS: A Constrained Monte Carlo Tree Search Framework for Mathematical Reasoning in Large Language Model
  7. 2025SIGMA: Refining Large Language Model Reasoning via Sibling-Guided Monte Carlo Augmentation
  8. 2024Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning