Data to 14 Oct 2026

Process Reward Models That Think

arXiv 2504.16828 ↗First posted 23 Apr 2025cs.LG cs.AI cs.CLcited by 118

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more

LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Step-by-step verifiers -- also known as process reward models (PRMs) -- are a key ingredient for test-time scaling. PRMs require step-level supervision, making them expensive to train. This work aims to build data-efficient PRMs as verbalized step-wise reward models that verify every step in the solution by generating a verification chain-of-thought (CoT). We propose ThinkPRM, a long CoT verifier fine-tuned on orders of magnitude fewer process labels than those required by discriminative PRMs. Our approach capitalizes on the inherent reasoning abilities of long CoT models, and outperforms LLM-as-a-Judge and discriminative verifiers -- using only 1% of the process labels in PRM800K -- across several challenging benchmarks. Specifically, ThinkPRM beats the baselines on ProcessBench, MATH-500, and AIME '24 under best-of-N selection and reward-guided search. In an out-of-domain evaluation on a subset of GPQA-Diamond and LiveCodeBench, our PRM surpasses discriminative verifiers trained on the full PRM800K by 8% and 4.5%, respectively. Lastly, under the same token budget, ThinkPRM scales up verification compute more effectively compared to LLM-as-a-Judge, outperforming it by 7.2% on a subset of ProcessBench. Our work highlights the value of generative, long CoT PRMs that can scale test-time compute for verification while requiring minimal supervision for training. Our code, data, and models are released at https://github.com/mukhal/thinkprm.

Key facts. Cited by 118 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 55 of its 73 references that are in the map. First posted on arXiv 23 Apr 2025.

Builds on

55 of 73 references.

Closest ancestors · 10

Same direction, newest first

Other references · 45

2020Language Models are Few-Shot Learnerscited by 32,419
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
2025s1: Simple test-time scalingcited by 1,341
2024Qwen2.5-Coder Technical Reportcited by 1,203

33 more

2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
2024Distilling System 2 into System 1cited by 159
2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 152
2024Generative Reward Modelscited by 148
2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 119
2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
2024Critique-out-Loud Reward Modelscited by 106
2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96
2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problemscited by 89
2024Small Language Models Need Strong Verifiers to Self-Correct Reasoningcited by 86
2024Self-Taught Evaluatorscited by 72
2025Self-rewarding correction for mathematical reasoningcited by 70
2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problemscited by 50
2024When is Tree Search Useful for LLM Planning? It Depends on the Discriminatorcited by 47
2023LitCab: Lightweight Language Model Calibration over Short- and Long-form Responsescited by 44
2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 42
2023GRACE: Discriminator-Guided Chain-of-Thought Reasoningcited by 35
2025Verdict: A Library for Scaling Judge-Time Computecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 50; 2026: 68.

118 papers.

Sort:

By area:

2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 97
2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learningcited by 70
2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 40
2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 35
2026Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Gradingcited by 31
2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
2025StepWiser: Stepwise Generative Judges for Wiser Reasoningcited by 28
2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 26
2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 25
2025Interleaved Reasoning for Large Language Models via Reinforcement Learningcited by 21

38 more

2025HeurAgenix: Leveraging LLMs for Solving Complex Combinatorial Optimization Challengescited by 21
2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 21
2025Robust Reward Modeling via Causal Rubricscited by 19
2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
2025E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testingcited by 17
2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
2025GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoningcited by 14
2026InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoningcited by 14
2025Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learnedcited by 13
2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
2025Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learningcited by 12
2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 12
2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
2026Spark: Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learningcited by 11
2026Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learningcited by 11
2025Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Taskscited by 10
2025Rethinking Reward Models for Multi-Domain Test-Time Scalingcited by 10
2025Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learningcited by 10
2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
2025Efficient PRM Training Data Synthesis via Formal Verificationcited by 8
2025Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervisioncited by 8
2025Trust but Verify! A Survey on Verification Design for Test-time Scalingcited by 8
2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
2025RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedbackcited by 7
2025Fin-PRM: A Domain-Specialized Process Reward Model for Financial Reasoning in Large Language Modelscited by 7
2026Evidence-Augmented Policy Optimization with Reward Co-Evolution for Long-Context Reasoningcited by 7
2026AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputscited by 7
2026REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Tracescited by 7
2025Reasoning Model is Stubborn: Diagnosing Instruction Overriding in Reasoning Modelscited by 6
2025SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learningcited by 6
2025On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inferencecited by 6
2026Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluationcited by 6
2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agentscited by 6
2025TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoningcited by 5
2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
2026Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verificationcited by 5
2026Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysiscited by 5
2025SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modellingcited by 4

50 most cited of 118 All 118 (JSON)

Similar, not cited

Close in wording, no citation either way