mlsanityData to 7 Oct 2026

The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measure

arXiv 2205.10047 ↗First posted 20 May 2022cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

The popular Proximal Policy Optimization (PPO) algorithm approximates the solution in a clipped policy space. Does there exist better policies outside of this space? By using a novel surrogate objective that employs the sigmoid function (which provides an interesting way of exploration), we found that the answer is ``YES'', and the better policies are in fact located very far from the clipped space. We show that PPO is insufficient in ``off-policyness'', according to an off-policy metric called DEON. Our algorithm explores in a much larger policy space than PPO, and it maximizes the Conservative Policy Iteration (CPI) objective better than PPO during training. To the best of our knowledge, all current PPO methods have the clipping operation and optimize in the clipped policy space. Our method is the first of this kind, which advances the understanding of CPI optimization and policy gradient methods. Code is available at https://github.com/raincchio/P3O.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 35 of its 67 references that are in the map. First posted on arXiv 20 May 2022.

Builds on

35 of 67 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2016OpenAI Gymcited by 1,684
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  12. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
23 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  3. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  4. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  5. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  6. 2017Parameter Space Noise for Explorationcited by 158
  7. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  8. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  9. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
  10. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
  11. 2020Mirror Descent Policy Optimizationcited by 102
  12. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  13. 2019Truly Proximal Policy Optimizationcited by 68
  14. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  15. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 59
  16. 2020Revisiting Design Choices in Proximal Policy Optimizationcited by 38
  17. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  18. 2019Trust Region-Guided Proximal Policy Optimizationcited by 36
  19. 2019Deep Learning of Preconditioners for Conjugate Gradient Solvers in Urban Water Related Problemscited by 19
  20. 2022You May Not Need Ratio Clipping in PPOcited by 19
  21. 2021Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learningcited by 15
  22. 2018QUOTA: The Quantile Option Architecture for Reinforcement Learningcited by 13
  23. 2018Qualitative Measurements of Policy Discrepancy for Return-Based Deep Q-Networkcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 3.

12 papers.

  1. 2025Soft Adaptive Policy Optimizationcited by 89
  2. 2025Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchronycited by 13
  3. 2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalizationcited by 8
  4. 2025It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RLcited by 6
  5. 2025Enhancing PPO with Trajectory-Aware Hybrid Policiescited by 4
  6. 2025Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Samplingcited by 1
  7. 2025Robust Remote Reinforcement Learning over Unreliable Communication Channels using Homomorphic State Encodingcited by 0
  8. 2025Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Spacecited by 0
  9. 2025Memory-Driven Self-Improvement for Decision Making with Large Language Modelscited by 0
  10. 2026ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithmcited by 0
  11. 2026Soft Sequence Policy Optimizationcited by 0
  12. 2026Smooth Gate Functions for Soft Advantage Policy Optimizationcited by 0