mlsanityData to 14 Oct 2026

R-PRM: Reasoning-Driven Process Reward Modeling

arXiv 2503.21295 ↗First posted 27 Mar 2025cs.CLcited by 38

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large language models (LLMs) inevitably make mistakes when performing step-by-step mathematical reasoning. Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step. However, existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy, which is further exacerbated by the scarcity of annotated data. To address these issues, we propose Reasoning-Driven Process Reward Modeling (R-PRM). First, we leverage stronger LLMs to generate seed data from limited annotations, effectively bootstrapping our model's reasoning capabilities and enabling comprehensive step-by-step evaluation. Second, we further enhance performance through preference optimization, without requiring additional annotated data. Third, we introduce inference-time scaling to fully harness the model's reasoning potential. Extensive experiments demonstrate R-PRM's effectiveness: on ProcessBench and PRMBench, it surpasses strong baselines by 11.9 and 8.5 points in F1 scores, respectively. When applied to guide mathematical reasoning, R-PRM achieves consistent accuracy improvements of over 8.5 points across six challenging datasets. Further analysis reveals that R-PRM exhibits more comprehensive evaluation and stronger generalization capabilities, thereby highlighting its significant potential.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 30 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

19 of 30 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Process-based Self-Rewarding Language Models5 Mar
  2. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  3. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models6 Jan
  4. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  5. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  7. 2023Let's Verify Step by Step31 May
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 10

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  9. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  10. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 20.

38 papers.

Sort:

By area:

  1. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 38
  2. 2025What Makes a Good Reasoning Chain? Uncovering Structural Patterns in Long Chain-of-Thought Reasoningcited by 21
  3. 2025RAFFLES: Reasoning-based Attribution of Faults for LLM Systemscited by 21
  4. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  5. 2025GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoningcited by 14
  6. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoningcited by 14
  7. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
  8. 2025HERMES: Towards Efficient and Verifiable Mathematical Reasoning in LLMscited by 11
  9. 2025Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervisioncited by 8
  10. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  11. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  12. 2026DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignmentcited by 7
26 more
  1. 2025PATS: Process-Level Adaptive Thinking Mode Switchingcited by 6
  2. 2025StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Modelscited by 6
  3. 2025SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learningcited by 6
  4. 2026SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestrationcited by 6
  5. 2026Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verificationcited by 5
  6. 2026Characterizing, Evaluating, and Optimizing Complex Reasoningcited by 5
  7. 2025SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modellingcited by 4
  8. 2025ToolPRM: Fine-Grained Inference Scaling of Structured Outputs for Function Callingcited by 4
  9. 2025OPV: Outcome-based Process Verifier for Efficient Long Chain-of-Thought Verificationcited by 4
  10. 2026Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Modelscited by 4
  11. 2026SDR: Set-Distance Rewards for Radiology Report Generationcited by 3
  12. 2025AURA: Affordance-Understanding and Risk-aware Alignment Technique for Large Language Modelscited by 2
  13. 2025Exploring Generative Process Reward Modeling for Semi-Structured Data: A Case Study of Table Question Answeringcited by 2
  14. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representationscited by 2
  15. 2026Pause or Fabricate? Training Language Models for Grounded Reasoningcited by 2
  16. 2026From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Groundingcited by 2
  17. 2026Process Supervision via Verbal Critique Improves Reasoning in Large Language Modelscited by 1
  18. 2026VeriBound: PAC-Bayesian Generalization Bounds for Process Reward Models Trained with Formal Verification Toolscited by 1
  19. 2026Reasoning-targeted Jailbreak Attacks on Large Reasoning Models via Semantic Triggers and Psychological Framingcited by 0
  20. 2026Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoningcited by 0
  21. 2026MARD: Mirror-Augmented Reasoning Distillation for Mechanism-Level Drug-Drug Interaction Predictioncited by 0
  22. 2026Reinforcement Learning without Ground-Truth Solutions can Improve LLMscited by 0
  23. 2026Offline-Online Curriculum RL for Multimodal Reasoningcited by 0
  24. 2026SAGE: Mitigating Long-Horizon Reasoning Biases via Topological Guidancecited by 0
  25. 2026Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoningcited by 0
  26. 2026Learning Process Rewards via Reasoning State Propagationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
  2. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  3. 2025VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
  4. 2026Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards
  5. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning
  6. 2025Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning
  7. 2026GR-Ben: A General Reasoning Benchmark for Evaluating Process Reward Models
  8. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning