mlsanityData to 7 Oct 2026

Better Process Supervision with Bi-directional Rewarding Signals

arXiv 2503.04618 ↗First posted 6 Mar 2025cs.CLcited by 12

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Process supervision, i.e., evaluating each step, is critical for complex large language model (LLM) reasoning and test-time searching with increased inference compute. Existing approaches, represented by process reward models (PRMs), primarily focus on rewarding signals up to the current step, exhibiting a one-directional nature and lacking a mechanism to model the distance to the final target. To address this problem, we draw inspiration from the A* algorithm, which states that an effective supervisory signal should simultaneously consider the incurred cost and the estimated cost for reaching the target. Building on this key insight, we introduce BiRM, a novel process supervision model that not only evaluates the correctness of previous steps but also models the probability of future success. We conduct extensive experiments on mathematical reasoning tasks and demonstrate that BiRM provides more precise evaluations of LLM reasoning steps, achieving an improvement of 3.1% on Gaokao2023 over PRM under the Best-of-N sampling method. Besides, in search-based strategies, BiRM provides more comprehensive guidance and outperforms ORM by 5.0% and PRM by 3.8% respectively on MATH-500.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 34 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

22 of 34 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  2. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models6 Jan
  3. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  4. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  6. 2023Let's Verify Step by Step31 May
  7. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  5. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  6. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  7. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  8. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 103
  9. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95
  10. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  11. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 84
  12. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
3 more
  1. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  2. 2025Examining False Positives under Inference Scaling for Mathematical Reasoningcited by 11
  3. 2025Scaling Flaws of Verifier-Guided Search in Mathematical Reasoningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 2.

12 papers.

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  3. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 37
  4. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 36
  5. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 11
  6. 2025GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoningcited by 10
  7. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  8. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
  9. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 5
  10. 2026Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Taskscited by 3
  11. 2026GRLO: Towards Generalizable Reinforcement Learning in Open-Ended Environments from Zerocited by 1
  12. 2025Parrot: A Training Pipeline Enhances Both Program CoT and Natural Language CoT for Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  2. 2025An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning
  3. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
  4. 2025The Bidirectional Process Reward Model
  5. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling
  6. 2026Process Supervision for Chain-of-Thought Reasoning via Monte Carlo Net Information Gain
  7. 2026Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning
  8. 2025R-PRM: Reasoning-Driven Process Reward Modeling