mlsanityData to 14 Oct 2026

RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts

arXiv 2607.18470 ↗First posted 20 Jul 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) has shown strong effectiveness in reinforcement learning from verifiable feedback, where sampled rollouts can be compared within a group using task-provided correctness signals. However, extending group-relative optimization beyond verifiable settings is challenging because success in many tasks is not captured by a single correctness criterion. We propose Reference-Relative Policy Optimization (RRPO), which generalizes GRPO by replacing direct correctness-based advantage construction with reference-relative contrastive comparisons. RRPO first uses stratified conditional rollouts to construct positive and negative anchor sets, and then trains a metric projection head with a set-contrastive objective to compare candidate rollouts against these anchors. The resulting alignment scores directly define contrastive advantages: during policy optimization, the projection head is frozen, and the scores are centered within each rollout group in a standard group-relative objective. We evaluate RRPO using anchor-based contrastive advantages throughout policy optimization, without relying on task ground-truth verifiers. Across verifiable reasoning, open-ended generation, and post-SFT settings, RRPO remains competitive with verifier-based optimization, improves over weakly supervised baselines, and provides additional gains after supervised fine-tuning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 35 references that are in the map. First posted on arXiv 20 Jul 2026.

Builds on

26 of 35 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking13 May
  2. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning8 Apr
  3. 2026WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning19 Feb
  4. 2026Native Reasoning Models: Training Language Models to Reason on Unverifiable Data12 Feb
  5. 2026Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clustering13 Jan
  6. 2025GRPO is Secretly a Process Reward Model25 Sep
  7. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  9. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  10. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2019ELI5: Long Form Question Answeringcited by 575
  6. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  7. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  8. 2026SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenescited by 11
  9. 2025Listwise Preference Diffusion Optimization for User Behavior Trajectories Predictioncited by 10
  10. 2025In-context Ranking Preference Optimizationcited by 9
  11. 2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Taskscited by 7
  12. 2026GOPO: Policy Optimization using Ranked Rewardscited by 5
2 more
  1. 2026OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agentscited by 2
  2. 2026MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  2. 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning
  3. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
  4. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  5. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
  6. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  7. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  8. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation