# On-Policy RL with Optimal Reward Baseline

arXiv: https://arxiv.org/abs/2505.23585 · HTML: https://mlsanity.com/paper/2505.23585/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 29 May 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models](https://mlsanity.com/paper/2310.10505/) (2023)
- [Understanding the impact of entropy on policy optimization](https://mlsanity.com/paper/1811.11214/) (2018)
- [The Optimal Reward Baseline for Gradient-Based Reinforcement Learning](https://mlsanity.com/paper/1301.2315/) (2013)

## Built on by (most cited first, 30 of 34)

- [Geometric-Mean Policy Optimization](https://mlsanity.com/paper/2507.20673/) (2025)
- [Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers](https://mlsanity.com/paper/2510.11370/) (2025)
- [Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective](https://mlsanity.com/paper/2510.10150/) (2025)
- [Reinforcement Pre-Training](https://mlsanity.com/paper/2506.08007/) (2025)
- [Single-stream Policy Optimization](https://mlsanity.com/paper/2509.13232/) (2025)
- [Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic](https://mlsanity.com/paper/2603.01162/) (2026)
- [DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training](https://mlsanity.com/paper/2604.26256/) (2026)
- [Scheduling Your LLM Reinforcement Learning with Reasoning Trees](https://mlsanity.com/paper/2510.24832/) (2025)
- [UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilities](https://mlsanity.com/paper/2507.19766/) (2025)
- [Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs](https://mlsanity.com/paper/2602.17616/) (2026)
- [When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO](https://mlsanity.com/paper/2603.13134/) (2026)
- [The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL](https://mlsanity.com/paper/2602.07078/) (2026)
- [V0.5: Generalist Value Model as a Prior for Sparse RL Rollouts](https://mlsanity.com/paper/2603.10848/) (2026)
- [Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR](https://mlsanity.com/paper/2605.05965/) (2026)
- [Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR](https://mlsanity.com/paper/2509.02522/) (2025)
- [Fast LLM Post-training via Decoupled and Fastest-of-N Speculation](https://mlsanity.com/paper/2511.16193/) (2025)
- [Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning](https://mlsanity.com/paper/2604.28005/) (2026)
- [Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization](https://mlsanity.com/paper/2605.11491/) (2026)
- [Policy Improvement Reinforcement Learning](https://mlsanity.com/paper/2604.00860/) (2026)
- [Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning](https://mlsanity.com/paper/2606.21943/) (2026)
- [ISOPO: Proximal policy gradients without pi-old](https://mlsanity.com/paper/2512.23353/) (2025)
- [Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMs](https://mlsanity.com/paper/2601.21476/) (2026)
- [One Ring to Rule Them All: Unifying Group-Based RL via Dynamic Power-Mean Geometry](https://mlsanity.com/paper/2601.22521/) (2026)
- [Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning](https://mlsanity.com/paper/2602.20197/) (2026)
- [TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesis](https://mlsanity.com/paper/2604.08468/) (2026)
- [BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning](https://mlsanity.com/paper/2605.27293/) (2026)
- [GAPO: Robust Advantage Estimation for Real-World Code LLMs](https://mlsanity.com/paper/2510.21830/) (2025)
- [Holder Policy Optimisation](https://mlsanity.com/paper/2605.12058/) (2026)
- [PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF](https://mlsanity.com/paper/2606.29758/) (2026)
- [Reinforcement Learning for Code Optimization](https://mlsanity.com/paper/2607.25970/) (2026)

## Abstract (arXiv)

Reinforcement learning algorithms are fundamental to align large language models with human preferences and to enhance their reasoning capabilities. However, current reinforcement learning algorithms often suffer from training instability due to loose on-policy constraints and computational inefficiency due to auxiliary models. In this work, we propose On-Policy RL with Optimal reward baseline (OPO), a novel and simplified reinforcement learning algorithm designed to address these challenges. OPO emphasizes the importance of exact on-policy training, which empirically stabilizes the training process and enhances exploration. Moreover, OPO integrates a practically feasible formulation of the optimal reward baseline that minimizes gradient variance. We evaluate OPO on mathematical reasoning benchmarks. The results demonstrate its superior performance and training stability without additional models or regularization terms. Furthermore, OPO achieves lower policy shifts and higher output entropy, encouraging more diverse and less repetitive responses. These results highlight OPO as a promising direction for stable and effective reinforcement learning in large language model alignment and reasoning tasks. The implementation is merged into the verl library at https://verl.readthedocs.io/en/latest/algo/opo.html.