mlsanityData to 14 Oct 2026

On-Policy RL with Optimal Reward Baseline

arXiv 2505.23585 ↗First posted 29 May 2025cs.LG cs.CLcited by 34

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning algorithms are fundamental to align large language models with human preferences and to enhance their reasoning capabilities. However, current reinforcement learning algorithms often suffer from training instability due to loose on-policy constraints and computational inefficiency due to auxiliary models. In this work, we propose On-Policy RL with Optimal reward baseline (OPO), a novel and simplified reinforcement learning algorithm designed to address these challenges. OPO emphasizes the importance of exact on-policy training, which empirically stabilizes the training process and enhances exploration. Moreover, OPO integrates a practically feasible formulation of the optimal reward baseline that minimizes gradient variance. We evaluate OPO on mathematical reasoning benchmarks. The results demonstrate its superior performance and training stability without additional models or regularization terms. Furthermore, OPO achieves lower policy shifts and higher output entropy, encouraging more diverse and less repetitive responses. These results highlight OPO as a promising direction for stable and effective reinforcement learning in large language model alignment and reasoning tasks. The implementation is merged into the verl library at https://verl.readthedocs.io/en/latest/algo/opo.html.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 35 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

23 of 35 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Learning to Reason under Off-Policy Guidance21 Apr
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  12. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
5 more
  1. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  2. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  3. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  4. 2018Understanding the impact of entropy on policy optimizationcited by 176
  5. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 85

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 23.

34 papers.

Sort:

By area:

  1. 2025Geometric-Mean Policy Optimizationcited by 111
  2. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 72
  3. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 52
  4. 2025Reinforcement Pre-Trainingcited by 40
  5. 2025Single-stream Policy Optimizationcited by 28
  6. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  7. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
  8. 2025Scheduling Your LLM Reinforcement Learning with Reasoning Treescited by 12
  9. 2025UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilitiescited by 11
  10. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 11
  11. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPOcited by 10
  12. 2026The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RLcited by 6
22 more
  1. 2026V0.5: Generalist Value Model as a Prior for Sparse RL Rolloutscited by 5
  2. 2026Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVRcited by 5
  3. 2025Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVRcited by 4
  4. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  5. 2026Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoningcited by 3
  6. 2026Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimizationcited by 3
  7. 2026Policy Improvement Reinforcement Learningcited by 2
  8. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  9. 2025ISOPO: Proximal policy gradients without pi-oldcited by 1
  10. 2026Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMscited by 1
  11. 2026One Ring to Rule Them All: Unifying Group-Based RL via Dynamic Power-Mean Geometrycited by 1
  12. 2026Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoningcited by 1
  13. 2026TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesiscited by 1
  14. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoningcited by 1
  15. 2025GAPO: Robust Advantage Estimation for Real-World Code LLMscited by 0
  16. 2026Holder Policy Optimisationcited by 0
  17. 2026PS-PPO: Prefix-Sampling PPO for Critic-Free RLHFcited by 0
  18. 2026Reinforcement Learning for Code Optimizationcited by 0
  19. 2026Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimizationcited by 0
  20. 2026Expected Reasoning-Step Return Unifies On-Policy Learning from Rewards and Teacherscited by 0
  21. 2026SAT-RL: Training Language Models for Logical Reasoning with Symbolic Solverscited by 0
  22. 2026Ascent Conditions for Length-Normalized Single-Rollout Policy Updatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlap
  2. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
  3. 2025COPO: Consistency-Aware Policy Optimization
  4. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
  5. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  6. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  7. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  8. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle