# Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning

arXiv: https://arxiv.org/abs/2509.26578 · HTML: https://mlsanity.com/paper/2509.26578/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 41 of its 49 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 30 of 41)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://mlsanity.com/paper/2501.07301/) (2025)
- [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2502.03373/) (2025)
- [rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking](https://mlsanity.com/paper/2501.04519/) (2025)
- [ProcessBench: Identifying Process Errors in Mathematical Reasoning](https://mlsanity.com/paper/2412.06559/) (2024)
- [Self-Evaluation Guided Beam Search for Reasoning](https://mlsanity.com/paper/2305.00633/) (2023)
- [Free Process Rewards without Process Labels](https://mlsanity.com/paper/2412.01981/) (2024)
- [GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers](https://mlsanity.com/paper/2402.19255/) (2024)
- [Active Prompting with Chain-of-Thought for Large Language Models](https://mlsanity.com/paper/2302.12246/) (2023)
- [RM-R1: Reward Modeling as Reasoning](https://mlsanity.com/paper/2505.02387/) (2025)
- [GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning](https://mlsanity.com/paper/2504.00891/) (2025)
- [On Designing Effective RL Reward at Training Time for LLM Reasoning](https://mlsanity.com/paper/2410.15115/) (2024)
- [OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning](https://mlsanity.com/paper/2311.09724/) (2023)
- [MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline](https://mlsanity.com/paper/2401.08190/) (2024)

## Built on by (most cited first, 14 of 14)

- [Exploring Reasoning Reward Model for Agents](https://mlsanity.com/paper/2601.22154/) (2026)
- [Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey](https://mlsanity.com/paper/2510.01925/) (2025)
- [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026)
- [DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignment](https://mlsanity.com/paper/2605.03327/) (2026)
- [Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning](https://mlsanity.com/paper/2602.01791/) (2026)
- [Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents](https://mlsanity.com/paper/2606.26080/) (2026)
- [Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimization](https://mlsanity.com/paper/2602.02545/) (2026)
- [IAPO: Information-Aware Policy Optimization for Token-Efficient Reasoning](https://mlsanity.com/paper/2602.19049/) (2026)
- [Epistemic Regret Minimization: Label-Free Causal Critique Beyond Outcome Reward](https://mlsanity.com/paper/2602.11675/) (2026)
- [From Correctness to Utility: Gain-Based Prefix Evaluation for LLM Reasoning](https://mlsanity.com/paper/2606.07190/) (2026)
- [Granularity-Regulated Adaptive Computational Efficiency for Optimal Verification in Test-Time Scaling](https://mlsanity.com/paper/2606.19354/) (2026)
- [VeriBound: PAC-Bayesian Generalization Bounds for Process Reward Models Trained with Formal Verification Tools](https://mlsanity.com/paper/2606.20740/) (2026)
- [What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels](https://mlsanity.com/paper/2608.22960/) (2026)
- [Learning Process Rewards via Reasoning State Propagation](https://mlsanity.com/paper/2609.39220/) (2026)

## Abstract (arXiv)

Process Reward Models (PRMs) have emerged as a promising approach to enhance the reasoning capabilities of large language models (LLMs) by guiding their step-by-step reasoning toward a final answer. However, existing PRMs either treat each reasoning step in isolation, failing to capture inter-step dependencies, or struggle to align process rewards with the final outcome. Consequently, the reward signal fails to respect temporal causality in sequential reasoning and faces ambiguous credit assignment. These limitations make downstream models vulnerable to reward hacking and lead to suboptimal performance. In this work, we propose Conditional Reward Modeling (CRM) that frames LLM reasoning as a temporal process leading to a correct answer. The reward of each reasoning step is not only conditioned on the preceding steps but also explicitly linked to the final outcome of the reasoning trajectory. By enforcing conditional probability rules, our design captures the causal relationships among reasoning steps, with the link to the outcome allowing precise attribution of each intermediate step, thereby resolving credit assignment ambiguity. Further, through this consistent probabilistic modeling, the rewards produced by CRM enable more reliable cross-sample comparison. Experiments across Best-of-N sampling, beam search and reinforcement learning demonstrate that CRM consistently outperforms existing reward models, offering a principled framework for enhancing LLM reasoning. In particular, CRM is more robust to reward hacking and delivers stable downstream improvements without relying on verifiable rewards derived from ground truth.