mlsanityData to 7 Oct 2026

Truly Proximal Policy Optimization

arXiv 1903.07940 ↗First posted 19 Mar 2019cs.LG cs.AI stat.MLcited by 68

Not on the map. Why?

Abstract (arXiv)

Proximal policy optimization (PPO) is one of the most successful deep reinforcement-learning methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, its optimization behavior is still far from being fully understood. In this paper, we show that PPO could neither strictly restrict the likelihood ratio as it attempts to do nor enforce a well-defined trust region constraint, which means that it may still suffer from the risk of performance instability. To address this issue, we present an enhanced PPO method, named Truly PPO. Two critical improvements are made in our method: 1) it adopts a new clipping function to support a rollback behavior to restrict the difference between the new policy and the old one; 2) the triggering condition for clipping is replaced with a trust region-based one, such that optimizing the resulted surrogate objective function provides guaranteed monotonic improvement of the ultimate policy performance. It seems, by adhering more truly to making the algorithm proximal - confining the policy within the trust region, the new algorithm improves the original PPO on both sample efficiency and performance.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 11; 2023: 12; 2024: 7; 2025: 13; 2026: 22.

68 papers.

Sort:

By area:

  1. 2022Discovered Policy Optimisationcited by 103
  2. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 57
  3. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 39
  4. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 32
  5. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  6. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  7. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  8. 2023Order Matters: Agent-by-agent Policy Optimizationcited by 24
  9. 2022You May Not Need Ratio Clipping in PPOcited by 19
  10. 2024Discovering Temporally-Aware Reinforcement Learning Algorithmscited by 18
  11. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Modelcited by 14
  12. 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
38 more
  1. 2022Heterogeneous-Agent Mirror Learning: A Continuum of Solutions to Cooperative MARLcited by 12
  2. 2024Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rankcited by 7
  3. 2025TROLL: Trust Regions improve Reinforcement Learning for Large Language Modelscited by 7
  4. 2025Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Modelscited by 7
  5. 2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
  6. 2025TensorRL-QAS: Reinforcement learning with tensor networks for improved quantum architecture searchcited by 6
  7. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  8. 2022Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratiocited by 5
  9. 2026Rethinking the Divergence Regularization in LLM RLcited by 5
  10. 2022Autonomous Drone Swarm Navigation and Multi-target Tracking in 3D Environments with Dynamic Obstaclescited by 4
  11. 2025PPO in the Fisher-Rao geometrycited by 4
  12. 2026ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learningcited by 4
  13. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 4
  14. 2022Good Time to Ask: A Learning Framework for Asking for Help in Embodied Visual Navigationcited by 3
  15. 2023Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimizationcited by 3
  16. 2025DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Modelscited by 3
  17. 2025Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectoriescited by 3
  18. 2026Clipping-Free Policy Optimization for Large Language Modelscited by 3
  19. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  20. 2026Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learningcited by 3
  21. 2023Deep Q-Learning versus Proximal Policy Optimization: Performance Comparison in a Material Sorting Taskcited by 2
  22. 2023Machine Learning Meets Advanced Robotic Manipulationcited by 2
  23. 2023Absolute Policy Optimizationcited by 2
  24. 2025BenchRL-QAS: Benchmarking reinforcement learning algorithms for quantum architecture searchcited by 2
  25. 2021CIM-PPO:Proximal Policy Optimization with Liu-Correntropy Induced Metriccited by 1
  26. 2022Proximal Policy Optimization Learning based Control of Congested Freeway Trafficcited by 1
  27. 2022Generalized Policy Improvement Algorithms with Theoretically Supported Sample Reusecited by 1
  28. 2023Robustness of Utilizing Feedback in Embodied Visual Navigationcited by 1
  29. 2023A Review of Reinforcement Learning for Natural Language Processing, and Applications in Healthcarecited by 1
  30. 2023Dropout Strategy in Reinforcement Learning: Limiting the Surrogate Objective Variance in Policy Optimization Methodscited by 1
  31. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimizationcited by 1
  32. 2026Align and Filter: Improving Performance in Asynchronous On-Policy RLcited by 1
  33. 2026GIPO: Gaussian Importance Sampling Policy Optimizationcited by 1
  34. 2026FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignmentcited by 1
  35. 2026A Deep Reinforcement Learning (DRL)-Based Transformer Method for Solving the Open Shop Scheduling Problemcited by 1
  36. 2022Entropy Augmented Reinforcement Learningcited by 0
  37. 2023Joint action loss for proximal policy optimizationcited by 0
  38. 2023Partial advantage estimator for proximal policy optimizationcited by 0

50 most cited of 68 All 68 (JSON)