mlsanityData to 14 Oct 2026

Beyond the Boundaries of Proximal Policy Optimization

arXiv 2411.00666 ↗First posted 1 Nov 2024cs.LG cs.AIcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Proximal policy optimization (PPO) is a widely-used algorithm for on-policy reinforcement learning. This work offers an alternative perspective of PPO, in which it is decomposed into the inner-loop estimation of update vectors, and the outer-loop application of updates using gradient ascent with unity learning rate. Using this insight we propose outer proximal policy optimization (outer-PPO); a framework wherein these update vectors are applied using an arbitrary gradient-based optimizer. The decoupling of update estimation and update application enabled by outer-PPO highlights several implicit design choices in PPO that we challenge through empirical investigation. In particular we consider non-unity learning rates and momentum applied to the outer loop, and a momentum-bias applied to the inner estimation loop. Methods are evaluated against an aggressively tuned PPO baseline on Brax, Jumanji and MinAtar environments; non-unity learning rates and momentum both achieve statistically significant improvement on Brax and Jumanji, given the same hyperparameter tuning budget.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 32 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

18 of 32 references.

References · 18

  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  2. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  3. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  6. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  7. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  8. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  9. 2023Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performancecited by 147
  10. 2022Discovered Policy Optimisationcited by 104
  11. 2023Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAXcited by 58
  12. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 49
6 more
  1. 2018GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithmscited by 40
  2. 2020Revisiting Design Choices in Proximal Policy Optimizationcited by 38
  3. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  4. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  5. 2021Correcting Momentum in Temporal Difference Learningcited by 12
  6. 2023ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantagescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  2. 2025Context-Aware Agentic Power Resources Optimisation in EV using Smart2ChargeAppcited by 0
  3. 2026Bounded Ratio Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithm
  2. 2017Proximal Policy Optimization Algorithms
  3. 2026KLip-PPO: A per-sample KL perspective on PPO-Clip
  4. 2024Transductive Off-policy Proximal Policy Optimization
  5. 2026An Approximate Ascent Approach To Prove Convergence of PPO
  6. 2025Enhancing PPO with Trajectory-Aware Hybrid Policies
  7. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization
  8. 2026Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?