mlsanityData to 14 Oct 2026

Limits of PRM-Guided Tree Search for Mathematical Reasoning with LLMs

arXiv 2510.20272 ↗First posted 23 Oct 2025cs.LG cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

While chain-of-thought prompting with Best-of-N (BoN) selection has become popular for mathematical reasoning in large language models (LLMs), its linear structure fails to capture the branching and exploratory nature of complex problem-solving. In this work, we propose an adaptive algorithm to maximize process reward model (PRM) scores over the intractable action space, and investigate whether PRM-guided tree search can improve mathematical reasoning by exploring multiple partial solution paths. Across 23 diverse mathematical problems using Qwen2.5-Math-7B-Instruct with its associated PRM as a case study, we find that: (1) PRM-guided tree search shows no statistically significant improvements over BoN despite higher costs, (2) Monte Carlo tree search and beam search outperform other PRM-guided tree search methods, (3) PRMs poorly approximate state values and their reliability degrades with reasoning depth, and (4) PRMs generalize poorly out of distribution. This underperformance stems from tree search's greater reliance on unreliable PRM scores, suggesting different reward modeling is necessary before tree search can effectively enhance mathematical reasoning in LLMs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 8 of its 27 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

8 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  3. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 5

  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  3. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  4. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 158
  5. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026VeriBound: PAC-Bayesian Generalization Bounds for Process Reward Models Trained with Formal Verification Toolscited by 1
  2. 2026Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guidecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  2. 2024Enhancing LLM Reasoning with Reward-guided Tree Search
  3. 2025Unifying Tree Search Algorithm and Reward Design for LLM Reasoning: A Survey
  4. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
  5. 2025Retrieval-Augmented Process Reward Model for Generalizable Mathematical Reasoning
  6. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling
  7. 2025Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNets
  8. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision