mlsanityData to 7 Oct 2026

ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithm

arXiv 2602.09726 ↗First posted 10 Feb 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning has been able to solve various tasks successfully, however, due to the construction of policy gradient and training dynamics, tuning deep reinforcement learning models remains challenging. As one of the most successful deep reinforcement-learning algorithm, the Proximal Policy Optimization algorithm (PPO) clips the policy gradient within a conservative on-policy updates, which ensures reliable and stable policy improvement. However, this training pattern may sacrifice sample efficiency. On the other hand, off-policy methods make more adequate use of data through sample reuse, though at the cost of increased the estimation variance and bias. To leverage the advantages of both, in this paper, we propose a new PPO variant based on the stability guarantee from conservative on-policy iteration with a more efficient off-policy data utilization. Specifically, we first derive an extended off-policy improvement from an expectation form of generalized policy improvement lower bound. Then, we extend the clipping mechanism with segmented exponential functions for a suitable surrogate objective function. Third, the trajectories generated by the past M policies are organized in the replay buffer for off-policy training. We refer to this method as Extended Off-policy Proximal Policy Optimization (ExO-PPO). Compared with PPO and some other state-of-the-art variants, we demonstrate an improved performance of ExO-PPO with balanced sample efficiency and stability on varied tasks in the empirical experiments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

32 of 47 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2016OpenAI Gymcited by 1,684
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  12. 2017Constrained Policy Optimizationcited by 928
20 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  4. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  5. 2019Soft Actor-Critic for Discrete Action Settingscited by 161
  6. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  7. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  8. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  9. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  10. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  11. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
  12. 2019Truly Proximal Policy Optimizationcited by 68
  13. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 58
  14. 2023Behavior Proximal Policy Optimizationcited by 43
  15. 2017Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learningcited by 40
  16. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 39
  17. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  18. 2019Trust Region-Guided Proximal Policy Optimizationcited by 36
  19. 2022You May Not Need Ratio Clipping in PPOcited by 19
  20. 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12

Built on by

Not cited yet.