mlsanityData to 14 Oct 2026

Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guide

arXiv 2608.30051 ↗First posted 30 Aug 2026cs.AI cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Process reward models (PRMs) provide dense step-level guidance for search-based reasoning, enabling inference-time compute to be allocated toward promising partial solutions. However, recent evidence suggests that PRM-guided search can over-optimize imperfect process rewards, pruning viable trajectories while expanding spurious ones. In this work, we theoretically show that directly leveraging PRM score is vulnerable to verifier noise through an extreme-value effect: non-viable prefixes become more likely to receive spuriously high scores as reasoning depth increase. Therefore, we formulate the PRM-guided search as a robust optimization problem over plausible reward perturbations, termed maximin PRM-guided search, leading to a training-free robust process supervision method that preserves promising alternatives when step-level scores are noisy. Maximin PRM-guided search mitigates this failure mode by reducing sensitivity to over-optimized PRM outliers. Without fine-tuning or online adaptation, maximin search consistently improves the PRM-guided search by 17-35% on average, outperforming outcome- and step-level baselines in 14 out of 16 settings. Our source code is available at https://github.com/tjoo512/maximin-search.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 63 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

50 of 63 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Limits of PRM-Guided Tree Search for Mathematical Reasoning with LLMs23 Oct
  2. 2025Process Reward Models That Think23 Apr
  3. 2025Process Reinforcement through Implicit Rewards3 Feb
  4. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  5. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  6. 2024AlphaMath Almost Zero: Process Supervision without Process6 May
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  8. 2023Let's Verify Step by Step31 May
  9. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 41

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
29 more
  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  2. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  4. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  5. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
  6. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  7. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  8. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  9. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  10. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 202
  11. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 196
  12. 2019Two models of double descent for weak featurescited by 195
  13. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  14. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  15. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  16. 2019Distributionally Robust Optimization: A Reviewcited by 76
  17. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 65
  18. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 39
  19. 2019More Data Can Hurt for Linear Regression: Sample-wise Double Descentcited by 37
  20. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  21. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
  22. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 25
  23. 2020Deterministic error bounds for kernel-based learning techniques under bounded noisecited by 24
  24. 2021Determinantal Beam Searchcited by 18
  25. 2021Robust Uncertainty Bounds in Reproducing Kernel Hilbert Spaces: A Convex Optimization Approachcited by 14
  26. 2025PRM-BAS: Enhancing Multimodal Reasoning through PRM-guided Beam Annealing Searchcited by 12
  27. 2025Safe exploration in reproducing kernel Hilbert spacescited by 10
  28. 2025Semantic-guided Diverse Decoding for Large Language Modelcited by 9
  29. 2025Enhancing Decision-Making of Large Language Models via Actor-Criticcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  2. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
  3. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  4. 2026The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning
  5. 2025Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision
  6. 2025R-PRM: Reasoning-Driven Process Reward Modeling
  7. 2026Process Rewards with Learned Reliability
  8. 2026SCI-PRM: A Tool Aware Process Reward Model for Scientific Reasoning Verification