Data to 7 Oct 2026

Free Process Rewards without Process Labels

arXiv 2412.01981 ↗First posted 2 Dec 2024cs.LG cs.CLcited by 162

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more

LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Different from its counterpart outcome reward models (ORMs), which evaluate the entire responses, a process reward model (PRM) scores a reasoning trajectory step by step, providing denser and more fine grained rewards. However, training a PRM requires labels annotated at every intermediate step, presenting significant challenges for both manual and automatic data collection. This paper aims to address this challenge. Both theoretically and empirically, we show that an textitimplicit PRM can be obtained at no additional cost, by simply training an ORM on the cheaper response-level labels. The only assumption is to parameterize the outcome reward as the log-likelihood ratios of the policy and reference models, which can be optimized regardless of the specific choice of loss objectives. In experiments, we instantiate our implicit PRMs with various objectives and evaluate their performance on MATH. We show that our implicit PRM outperforms a strong MCTS-based baseline textit'a la Math-Shepherd using less than 1/38 of the training data. Its performance can be further improved with majority voting. We further find that scaling up instructions and responses benefits our implicit PRM, and the latter brings a larger gain. Particularly, we find that our implicit PRM, when instantiated with the cross-entropy (CE) loss, is more data-efficient and can keep improving generation models even when trained with only one response per instruction, the setup that suffers from extreme data scarcity and imbalance. Further, instructions should be relevant to downstream tasks while the diversity of responses does not bring gains. Surprisingly, training on extra Math-Shepherd step labels brings no further improvements to our implicit PRM trained on only outcome data. We hope that our work will encourage a rethinking of PRM training approaches and contribute to making training PRMs more accessible.

Key facts. Cited by 162 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 3 of its 47 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

3 of 47 references.

Closest ancestors · 1

Same direction, newest first

2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct

Other references · 2

2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 105; 2026: 57.

162 papers.

Sort:

By area:

2025Process Reinforcement through Implicit Rewardscited by 478
2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
2025TTRL: Test-Time Reinforcement Learningcited by 253
2025Reasoning Models Can Be Effective Without Thinkingcited by 182
2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124

38 more

2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
2025Process Reward Models That Thinkcited by 118
2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 104
2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitatecited by 61
2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 58
2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
20251.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Trainingcited by 49
2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoningcited by 49
2025ExGRPO: Learning to Reason from Experiencecited by 49
2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learningcited by 48
2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Rewardcited by 45
2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
2025Reasoning Language Models: A Blueprintcited by 41
2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 41
2025Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokenscited by 41
2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 33
2025AM-Thinking-v1: Advancing the Frontier of Reasoning at 32B Scalecited by 29
2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
2025AgentRM: Enhancing Agent Generalization with Reward Modelingcited by 25
2025Not All Correct Answers Are Equal: Why Your Distillation Source Matterscited by 25
2025AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidencecited by 23
2025Table-R1: Inference-Time Scaling for Table Reasoningcited by 23
2025ProgRM: Build Better GUI Agents with Progress Rewardscited by 21
2025What Makes a Good Reasoning Chain? Uncovering Structural Patterns in Long Chain-of-Thought Reasoningcited by 21
2025Shrinking the Generation-Verification Gap with Weak Verifierscited by 21
2025QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Searchcited by 20
2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Trainingcited by 17
2026ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agentscited by 17

50 most cited of 162 All 162 (JSON)

Similar, not cited

Close in wording, no citation either way