mlsanityData to 14 Oct 2026

GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning

arXiv 2504.00891 ↗First posted 1 Apr 2025cs.CLcited by 97

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have shown that it is promising to utilize Process Reward Models (PRMs) as verifiers to enhance the performance of LLMs. However, current PRMs face three key challenges: (1) limited process supervision and generalization capabilities, (2) dependence on scalar value prediction without leveraging the generative abilities of LLMs, and (3) inability to scale the test-time compute of PRMs. In this work, we introduce GenPRM, a generative process reward model that performs explicit Chain-of-Thought (CoT) reasoning with code verification before providing judgment for each reasoning step. To obtain high-quality process supervision labels and rationale data, we propose Relative Progress Estimation (RPE) and a rationale synthesis framework that incorporates code verification. Experimental results on ProcessBench and several mathematical reasoning tasks show that GenPRM significantly outperforms prior PRMs with only 23K training data from MATH dataset. Through test-time scaling, a 1.5B GenPRM outperforms GPT-4o, and a 7B GenPRM surpasses Qwen2.5-Math-PRM-72B on ProcessBench. Additionally, GenPRM demonstrates strong abilities to serve as a critic model for policy model refinement. This work establishes a new paradigm for process supervision that bridges the gap between PRMs and critic models in LLMs. Our code, model, and data will be available in https://ryanliu112.github.io/GenPRM.

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 53 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

42 of 53 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Retrieval-Augmented Process Reward Model for Generalizable Mathematical Reasoning20 Feb
  2. 2025AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verification17 Feb
  3. 2025Process Reward Models for LLM Agents: Practical Framework and Directions14 Feb
  4. 2025VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data10 Feb
  5. 2025Process Reinforcement through Implicit Rewards3 Feb
  6. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  7. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models6 Jan
  8. 2024Entropy-Regularized Process Reward Model15 Dec
  9. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  10. 2024Free Process Rewards without Process Labels2 Dec
  11. 2024Process Reward Model with Q-Value Rankings15 Oct
  12. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
5 more
  1. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  2. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  4. 2023Let's Verify Step by Step31 May
  5. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  12. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
13 more
  1. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  2. 2024Generative Reward Modelscited by 148
  3. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  4. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 139
  5. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 136
  6. 2024Critique-out-Loud Reward Modelscited by 106
  7. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 94
  8. 2024o1-Coder: an o1 Replication for Codingcited by 78
  9. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 42
  10. 2023PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuningcited by 29
  11. 2025MedS3: Towards Medical Slow Thinking with Self-Evolved Soft Dual-sided Process Supervisioncited by 24
  12. 2023PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulationcited by 9
  13. 2025Dyve: Thinking Fast and Slow for Dynamic Process Verificationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 52; 2026: 45.

97 papers.

Sort:

By area:

  1. 2025TTRL: Test-Time Reinforcement Learningcited by 275
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
  4. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 56
  5. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 40
  6. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVRcited by 39
  7. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 38
  8. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 35
  9. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  10. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  11. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 28
  12. 2025StepWiser: Stepwise Generative Judges for Wiser Reasoningcited by 28
38 more
  1. 2025Heimdall: test-time scaling on the generative verificationcited by 26
  2. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL Systemcited by 24
  3. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 20
  4. 2025Robust Reward Modeling via Causal Rubricscited by 19
  5. 2025MUR: Momentum Uncertainty guided Reasoning for Large Language Modelscited by 17
  6. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
  7. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  8. 2025GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoningcited by 14
  9. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoningcited by 14
  10. 2026ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agentscited by 14
  11. 2025Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMscited by 13
  12. 2025Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learnedcited by 13
  13. 2025Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Modelscited by 13
  14. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
  15. 2025VRPRM: Process Reward Modeling via Visual Reasoningcited by 11
  16. 2025VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detectioncited by 11
  17. 2026Reward Modeling from Natural Language Human Feedbackcited by 11
  18. 2026Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learningcited by 11
  19. 2025SPARK: Synergistic Policy And Reward Co-Evolving Frameworkcited by 10
  20. 2025Rethinking Reward Models for Multi-Domain Test-Time Scalingcited by 10
  21. 2025Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learningcited by 10
  22. 2025ReProbe: Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Modelscited by 10
  23. 2026V-Zero: Self-Improving Multimodal Reasoning with Zero Annotationcited by 10
  24. 2026LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewardscited by 10
  25. 2025LaSeR: Reinforcement Learning with Last-Token Self-Rewardingcited by 9
  26. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  27. 2025Solve-Detect-Verify: Inference-Time Scaling with Flexible Generative Verifiercited by 8
  28. 2025Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervisioncited by 8
  29. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  30. 2025SSR: Socratic Self-Refine for Large Language Model Reasoningcited by 7
  31. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  32. 2025SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learningcited by 6
  33. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  34. 2025Rethinking Optimal Verification Granularity for Compute-Efficient Test-Time Scalingcited by 5
  35. 2025Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patternscited by 5
  36. 2025CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoningcited by 5
  37. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  38. 2025TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoningcited by 5

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
  2. 2025R-PRM: Reasoning-Driven Process Reward Modeling
  3. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  4. 2025Process Reward Models That Think
  5. 2025GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning
  6. 2026GR-Ben: A General Reasoning Benchmark for Evaluating Process Reward Models
  7. 2026FunPRM: Function-as-Step Process Reward Model with Meta Reward Correction for Code Generation
  8. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling