mlsanityData to 14 Oct 2026

PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization

arXiv 2601.07182 ↗First posted 12 Jan 2026cs.LG cs.AIcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Policy optimization for large language models often suffers from sparse reward signals in multi-step reasoning tasks. Critic-free methods like GRPO assign a single normalized outcome reward to all tokens, providing limited guidance for intermediate reasoning . While Process Reward Models (PRMs) offer dense feedback, they risk premature collapse when used alone, as early low-reward tokens can drive policies toward truncated outputs. We introduce Process Relative Policy Optimization (PRPO), which combines outcome reliability with process-level guidance in a critic-free framework. PRPO segments reasoning sequences based on semantic clues, normalizes PRM scores into token-level advantages, and aligns their distribution with outcome advantages through location-parameter shift. On MATH500, PRPO improves Qwen2.5-Math-1.5B accuracy from 61.2% to 64.4% over GRPO using only eight rollouts and no value network, demonstrating efficient fine-grained credit assignment within critic-free optimization. Code is available at: https://github.com/SchumiDing/srpocode

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 24 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

19 of 24 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning21 Apr
  2. 2025More Bang for the Buck: Process Reward Modeling with Entropy-Driven Uncertainty28 Mar
  3. 2025Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning20 Feb
  4. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  5. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models6 Jan
  6. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs26 Jun
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec

Other references · 12

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  5. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  6. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  7. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  8. 2025TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimizationcited by 49
  9. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 36
  10. 2024Process Supervision-Guided Policy Optimization for Code Generationcited by 26
  11. 2025RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Trainingcited by 12
  12. 2025Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihoodcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Does Your Reasoning Model Implicitly Know When to Stop Thinking?cited by 19
  2. 2026Real-Time Aligned Reward Model beyond Semanticscited by 13
  3. 2026When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Modelscited by 6
  4. 2026One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinementcited by 6
  5. 2026SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregationcited by 3
  6. 2026StainFlow: Entity-Stain Tracking and Evidence Linking for Process Rewards in GUI Agentscited by 2
  7. 2026Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GRPO is Secretly a Process Reward Model
  2. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  3. 2024PSPO*: An Effective Process-supervised Policy Optimization for Reasoning Alignment
  4. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
  5. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
  6. 2026BoT-GRPO: Efficient Process-Reward RL for Reasoning via Bag-of-Token Aggregation
  7. 2026Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning
  8. 2026Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals