mlsanityData to 14 Oct 2026

Revisiting Design Choices in Proximal Policy Optimization

arXiv 2009.10897 ↗First posted 23 Sep 2020cs.LG stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

Proximal Policy Optimization (PPO) is a popular deep policy gradient algorithm. In standard implementations, PPO regularizes policy updates with clipped probability ratios, and parameterizes policies with either continuous Gaussian distributions or discrete Softmax distributions. These design choices are widely accepted, and motivated by empirical performance comparisons on MuJoCo and Atari benchmarks. We revisit these practices outside the regime of current benchmarks, and expose three failure modes of standard PPO. We explain why standard design choices are problematic in these cases, and show that alternative choices of surrogate objectives and policy parameterizations can prevent the failure modes. We hope that our work serves as a reminder that many algorithmic design choices in reinforcement learning are tied to specific simulation environments. We should not implicitly accept these choices as a standard part of a more general algorithm.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 12; 2023: 4; 2024: 6; 2025: 8; 2026: 6.

38 papers.

Sort:

By area:

  1. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 150
  2. 2022Discovered Policy Optimisationcited by 104
  3. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  4. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
  5. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 61
  6. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  7. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  8. 2021A general class of surrogate functions for stable and efficient reinforcement learningcited by 18
  9. 2022Lexicographic Multi-Objective Reinforcement Learningcited by 17
  10. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 16
  11. 2024Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibrationcited by 13
  12. 2025Reevaluating Policy Gradient Methods for Imperfect-Information Gamescited by 13
26 more
  1. 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
  2. 2022Heterogeneous-Agent Mirror Learning: A Continuum of Solutions to Cooperative MARLcited by 12
  3. 2025CPGD: Toward Stable Rule-based Reinforcement Learning for Language Modelscited by 11
  4. 2022Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimizationcited by 10
  5. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
  6. 2023Uncertainty Estimation for Safety-critical Scene Segmentation via Fine-grained Reward Maximizationcited by 7
  7. 2021Learning to Design and Construct Bridge without Blueprintcited by 6
  8. 2022RMBench: Benchmarking Deep Reinforcement Learning for Robotic Manipulator Controlcited by 5
  9. 2022Theta-Resonance: A Single-Step Reinforcement Learning Method for Design Space Explorationcited by 4
  10. 2024Beyond the Boundaries of Proximal Policy Optimizationcited by 3
  11. 2022Teacher-student curriculum learning for reinforcement learningcited by 2
  12. 2022Learning to Schedule Heuristics for the Simultaneous Stochastic Optimization of Mining Complexescited by 1
  13. 2023A Statistical Guarantee for Representation Transfer in Multitask Imitation Learningcited by 1
  14. 2023Guaranteed Trust Region Optimization via Two-Phase KL Penalizationcited by 1
  15. 2024Extremum-Seeking Action Selection for Accelerating Policy Optimizationcited by 1
  16. 2024Context Filtering with Reward Modeling in Question Answeringcited by 1
  17. 2025ULTHO: Ultra-Lightweight yet Efficient Hyperparameter Optimization in Deep Reinforcement Learningcited by 1
  18. 2025CORE: Constraint-Aware One-Step Reinforcement Learning for Simulation-Guided Neural Network Accelerator Designcited by 1
  19. 2022Entropy Augmented Reinforcement Learningcited by 0
  20. 2025RankPO: Preference Optimization for Job-Talent Matchingcited by 0
  21. 2025Inverse Design in Distributed Circuits Using Single-Step Reinforcement Learningcited by 0
  22. 2026Quantum-Inspired Reinforcement Learning for Secure and Sustainable AIoT-Driven Supply Chain Systemscited by 0
  23. 2026Demystifying Design Choices of Reinforcement Fine-tuning: A Batched Contextual Bandit Learning Perspectivecited by 0
  24. 2026Fine-tuning Timeseries Predictors Using Reinforcement Learningcited by 0
  25. 2026LogNEO: A GPT-Neo Reinforcement Learning Framework for Accurate Real-Time Log Anomaly Detectioncited by 0
  26. 2026KLip-PPO: A per-sample KL perspective on PPO-Clipcited by 0