mlsanityData to 14 Oct 2026

Improving Value-based Process Verifier via Structural Prior Injection

arXiv 2502.17498 ↗First posted 21 Feb 2025cs.LG cs.AI cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

In the Large Language Model(LLM) reasoning scenario, people often estimate state value via Monte Carlo sampling. Though Monte Carlo estimation is an elegant method with less inductive bias, noise and errors are inevitably introduced due to the limited sampling. To handle the problem, we inject the structural prior into the value representation and transfer the scalar value into the expectation of a pre-defined categorical distribution, representing the noise and errors from a distribution perspective. Specifically, by treating the result of Monte Carlo sampling as a single sample from the prior ground-truth Binomial distribution, we quantify the sampling error as the mismatch between posterior estimated distribution and ground-truth distribution, which is thus optimized via distribution selection optimization. We test the performance of value-based process verifiers on Best-of-N task and Beam search task. Compared with the scalar value representation, we show that reasonable structural prior injection induced by different objective functions or optimization methods can improve the performance of value-based process verifiers for about 1~2 points at little-to-no cost. We also show that under different structural prior, the verifiers' performances vary greatly despite having the same optimal solution, indicating the importance of reasonable structural prior injection.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 27 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

24 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  2. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  3. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning8 Feb
  4. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  5. 2023Let's Verify Step by Step31 May

Other references · 19

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2024Qwen2 Technical Reportcited by 2,120
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  10. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  11. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  12. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 112
7 more
  1. 2024Simplifying Deep Temporal Difference Learningcited by 78
  2. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  3. 2018Improving Regression Performance with Distributional Lossescited by 67
  4. 2023Improving Deep Regression with Ordinal Entropycited by 45
  5. 2023Automatic Model Selection with Large Language Models for Reasoningcited by 35
  6. 2024AutoPSV: Automated Process-Supervised Verifiercited by 30
  7. 2023Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughtscited by 27

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Value-based Process Verifier via Low-Cost Variance Reduction
  2. 2025Scaling Flaws of Verifier-Guided Search in Mathematical Reasoning
  3. 2024LLM2: Let Large Language Models Harness System 2 Reasoning
  4. 2025Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision
  5. 2025Taming Imperfect Process Verifiers: A Sampling Perspective on Backtracking
  6. 2025Know What You Don't Know: Uncertainty Calibration of Process Reward Models
  7. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening
  8. 2025Robust Search with Uncertainty-Aware Value Models for Language Model Reasoning