mlsanityData to 7 Oct 2026

P3O: Policy-on Policy-off Policy Optimization

arXiv 1905.01756 ↗First posted 5 May 2019cs.LG stat.MLcited by 37

Not on the map. Why?

Abstract (arXiv)

On-policy reinforcement learning (RL) algorithms have high sample complexity while off-policy algorithms are difficult to tune. Merging the two holds the promise to develop efficient algorithms that generalize across diverse environments. It is however challenging in practice to find suitable hyper-parameters that govern this trade off. This paper develops a simple algorithm named P3O that interleaves off-policy updates with on-policy updates. P3O uses the effective sample size between the behavior policy and the target policy to control how far they can be from each other and does not introduce any additional hyper-parameters. Extensive experiments on the Atari-2600 and MuJoCo benchmark suites show that this simple technique is effective in reducing the sample complexity of state-of-the-art algorithms. Code to reproduce experiments in this paper is at https://github.com/rasoolfa/P3O.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 9; 2022: 5; 2023: 4; 2024: 5; 2025: 6; 2026: 7.

37 papers.

Sort:

By area:

  1. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  2. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  3. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 39
  4. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Trainingcited by 33
  5. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generationcited by 29
  6. 2021Continuous Doubly Constrained Batch Reinforcement Learningcited by 27
  7. 2021Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmarkcited by 23
  8. 2024SAPG: Split and Aggregate Policy Gradientscited by 17
  9. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  10. 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
  11. 2022Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challengescited by 10
  12. 2024AlphaRouter: Quantum Circuit Routing with Reinforcement Learning and Tree Searchcited by 8
25 more
  1. 2021Faster Deep Reinforcement Learning with Slower Online Networkcited by 7
  2. 2023On-Policy Policy Gradient Reinforcement Learning Without On-Policy Samplingcited by 6
  3. 2021Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learningcited by 5
  4. 2021Adaptively Calibrated Critic Estimates for Deep Reinforcement Learningcited by 5
  5. 2022Time-Varying Propensity Score to Bridge the Gap between the Past and Presentcited by 5
  6. 2022Learning to Constrain Policy Optimization with Virtual Trust Regioncited by 4
  7. 2023Budgeting Counterfactual for Offline RLcited by 4
  8. 2025Enhancing PPO with Trajectory-Aware Hybrid Policiescited by 4
  9. 2025Offline Learning and Forgetting for Reasoning with Large Language Modelscited by 4
  10. 2024Revisiting Experience Replayable Conditionscited by 3
  11. 2021Cautious Actor-Criticcited by 2
  12. 2025Safety-Aware Reinforcement Learning for Control via Risk-Sensitive Action-Value Iteration and Quantile Regressioncited by 2
  13. 2021Coordinate-wise Control Variates for Deep Policy Gradientscited by 1
  14. 2022Generalized Policy Improvement Algorithms with Theoretically Supported Sample Reusecited by 1
  15. 2023Revisiting Estimation Bias in Policy Gradients for Deep Reinforcement Learningcited by 1
  16. 2023Distillation Policy Optimizationcited by 1
  17. 2024Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Reviewcited by 1
  18. 2024Learning the Target Network in Function Spacecited by 1
  19. 2026Align and Filter: Improving Performance in Asynchronous On-Policy RLcited by 1
  20. 2026ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithmcited by 0
  21. 2026Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updatescited by 0
  22. 2026MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidancecited by 0
  23. 2026Bounded Ratio Reinforcement Learningcited by 0
  24. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Trainingcited by 0
  25. 2026Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?cited by 0