mlsanityData to 14 Oct 2026

GOPO: Policy Optimization using Ranked Rewards

arXiv 2602.03876 ↗First posted 1 Feb 2026cs.LG cs.AIcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Standard reinforcement learning from human feedback (RLHF) trains a reward model on pairwise preference data and then uses it for policy optimization. However, while reward models are optimized to capture relative preferences, existing policy optimization techniques rely on absolute reward magnitudes during training. In settings where the rewards are non-verifiable such as summarization, instruction following, and chat completion, this misalignment often leads to suboptimal performance. We introduce Group Ordinal Policy Optimization (GOPO), a policy optimization method that uses only the ranking of the rewards and discards their magnitudes. Our rank-based transformation of rewards provides several gains, compared to Group Relative Policy Optimization (GRPO), in settings with non-verifiable rewards: (1) consistently higher training/validation reward trajectories, (2) improved LLM-as-judge evaluations across most intermediate training steps, and (3) reaching a policy of comparable quality in substantially less training steps than GRPO. We demonstrate consistent improvements across a range of tasks and model sizes.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 29 references that are in the map. First posted on arXiv 1 Feb 2026.

Builds on

19 of 29 references.

References · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2024A Survey on LLM-as-a-Judgecited by 1,423
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
7 more
  1. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  3. 2015Contextual Dueling Banditscited by 101
  4. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 60
  5. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 36
  6. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
  7. 2025Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocationcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 1
  2. 2026RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rolloutscited by 0
  3. 2026Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Constructioncited by 0
  4. 2026MatrixReward: Reward from Rubric Matrix for Open-Ended Generationcited by 0
  5. 2026Coding-Agent Benchmarks Should Match Their Users' Task Flowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
  2. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
  3. 2026Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment
  4. 2026Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals
  5. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback
  6. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  7. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  8. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences