mlsanityData to 14 Oct 2026

Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training

arXiv 2509.03403 ↗First posted 3 Sep 2025cs.LG cs.AIcited by 29

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) improves final-answer accuracy on reasoning tasks, but it does not reliably improve reasoning quality. Because outcome rewards only assess final answers, they also reward spurious successes: flawed reasoning can still receive maximal reward when it accidentally reaches the correct outcome. This outcome reward hacking creates biased gradients, making current RLVR insufficient for learning faithful reasoning. Process Reward Models (PRMs) provide step-wise supervision, but directly optimizing PRMs or naively combining them with outcome rewards is unstable under distribution shift during RL training process. We introduce PRocess cOnsistency Filter (PROF), a data curation method that uses PRM--ORM consistency for sample selection rather than direct reward optimization. PROF keeps correct responses with strong process support and incorrect responses with weak process support while maintaining a balanced training ratio. Experiments show that PROF consistently improves both final-answer accuracy and intermediate reasoning quality over strong baselines, with less dependence on strong PRMs.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 52 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

43 of 52 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs23 Jun
  2. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning21 May
  3. 2025Process Reward Models That Think23 Apr
  4. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning1 Apr
  5. 2025Process Reinforcement through Implicit Rewards3 Feb
  6. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  7. 2024Building Math Agents with Multi-Turn Iterative Preference Learning4 Sep
  8. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec

Other references · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
23 more
  1. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  4. 2024Self-Rewarding Language Modelscited by 681
  5. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  6. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  7. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
  8. 2023Faithful Chain-of-Thought Reasoningcited by 288
  9. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  10. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  11. 2023Mitigating the Alignment Tax of RLHFcited by 183
  12. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 145
  13. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  14. 2025Self-rewarding correction for mathematical reasoningcited by 70
  15. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chainscited by 65
  16. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 53
  17. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  18. 2020Understanding Learned Reward Functionscited by 37
  19. 2024Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphscited by 32
  20. 2024How Interpretable are Reasoning Explanations from Prompting Large Language Models?cited by 29
  21. 2025StepWiser: Stepwise Generative Judges for Wiser Reasoningcited by 28
  22. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 21
  23. 2025R.I.P.: Better Models by Survival of the Fittest Promptscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 20.

29 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 38
  3. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  4. 2025Cognitive Foundations for Reasoning and Their Manifestation in LLMscited by 26
  5. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 25
  6. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 16
  7. 2025Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Modelscited by 13
  8. 2025Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learningcited by 13
  9. 2026Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoningcited by 11
  10. 2025HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generationcited by 10
  11. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Modelcited by 10
  12. 2026SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewardscited by 9
17 more
  1. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  2. 2026DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignmentcited by 7
  3. 2026TreeAdv: Tree-Structured Advantage Redistribution for Group-Based RLcited by 5
  4. 2026HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learningcited by 4
  5. 2026MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loopcited by 4
  6. 2026Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidancecited by 3
  7. 2026LLM Reasoning with Process Rewards for Outcome-Guided Stepscited by 3
  8. 2026Process Supervision of Confidence Margin for Calibrated LLM Reasoningcited by 2
  9. 2026SAAS: Self-Aware Reinforcement Learning for Over-Search Mitigation in Agentic Searchcited by 2
  10. 2026Process Advantage Signal Shaping: A Paradigm-Agnostic Middleware for Process-Supervised RL in LLM Reasonerscited by 2
  11. 2026Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoningcited by 1
  12. 2026A Primer in Post-Training Reasoning Data: What We Know About How It Workscited by 1
  13. 2026Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learningcited by 1
  14. 2026Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVRcited by 1
  15. 2025RLFR: Extending Reinforcement Learning for LLMs with Flow Environmentcited by 0
  16. 2026PubSwap: Public-Data Off-Policy Coordination for Federated RLVRcited by 0
  17. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  2. 2025CLARity: Reasoning Consistency Alone Can Teach Reinforced Experts
  3. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  4. 2026Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models
  5. 2026SCOPE-RL: Optimizing Reasoning Paths Before and After Success
  6. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  7. 2026Beyond Correctness: Learning Robust Reasoning via Transfer
  8. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning