mlsanityData to 7 Oct 2026

Trust Region-Guided Proximal Policy Optimization

arXiv 1901.10314 ↗First posted 29 Jan 2019cs.LG cs.AI stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

Proximal policy optimization (PPO) is one of the most popular deep reinforcement learning (RL) methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, as a model-free RL method, the success of PPO relies heavily on the effectiveness of its exploratory policy search. In this paper, we give an in-depth analysis on the exploration behavior of PPO, and show that PPO is prone to suffer from the risk of lack of exploration especially under the case of bad initialization, which may lead to the failure of training or being trapped in bad local optima. To address these issues, we proposed a novel policy optimization method, named Trust Region-Guided PPO (TRGPPO), which adaptively adjusts the clipping range within the trust region. We formally show that this method not only improves the exploration ability within the trust region but enjoys a better performance bound compared to the original PPO as well. Extensive experiments verify the advantage of the proposed method.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 6; 2023: 2; 2024: 3; 2025: 6; 2026: 14.

36 papers.

Sort:

By area:

  1. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 39
  2. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 32
  3. 2021Differentiable Trust Region Layers for Deep Reinforcement Learningcited by 24
  4. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Controlcited by 18
  5. 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
  6. 2022Spatiotemporal Costmap Inference for MPC via Deep Inverse Reinforcement Learningcited by 10
  7. 2021Faster Deep Reinforcement Learning with Slower Online Networkcited by 7
  8. 2024Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rankcited by 7
  9. 2025TROLL: Trust Regions improve Reinforcement Learning for Large Language Modelscited by 7
  10. 2021Decaying Clipping Range in Proximal Policy Optimizationcited by 6
  11. 2026Rethinking the Divergence Regularization in LLM RLcited by 5
  12. 2022Learning to Constrain Policy Optimization with Virtual Trust Regioncited by 4
24 more
  1. 2024Safe and Robust Reinforcement Learning: Principles and Practicecited by 4
  2. 2026BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learningcited by 3
  3. 2026Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learningcited by 3
  4. 2025Overcoming Non-stationary Dynamics with Evidential Proximal Policy Optimizationcited by 2
  5. 2021Curious Explorer: a provable exploration strategy in Policy Learningcited by 1
  6. 2022Generalized Policy Improvement Algorithms with Theoretically Supported Sample Reusecited by 1
  7. 2024Distributed Stochastic Gradient Descent with Staleness: A Stochastic Delay Differential Equation Based Frameworkcited by 1
  8. 2025Stabilizing Policy Gradient Methods via Reward Profilingcited by 1
  9. 2026ETR: Outcome-Guided Elastic Trust Regions for Policy Optimizationcited by 1
  10. 2026Plasticity-Enhanced Multi-Agent Mixture of Experts for Dynamic Objective Adaptation in UAVs-Assisted Emergency Communication Networkscited by 1
  11. 2022Processing Network Controls via Deep Reinforcement Learningcited by 0
  12. 2022Entropy Augmented Reinforcement Learningcited by 0
  13. 2023Joint action loss for proximal policy optimizationcited by 0
  14. 2023A dynamical clipping approach with task feedback for Proximal Policy Optimizationcited by 0
  15. 2025Escaping Local Optima in the Waddington Landscape: A Two-Stage TRPO-PPO Approach for Single-Cell Perturbation Analysiscited by 0
  16. 2025Directional-Clamp PPOcited by 0
  17. 2026A Unified Framework for Rethinking Policy Divergence Measures in GRPOcited by 0
  18. 2026ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithmcited by 0
  19. 2026Bounded Ratio Reinforcement Learningcited by 0
  20. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  21. 2026KLip-PPO: A per-sample KL perspective on PPO-Clipcited by 0
  22. 2026Predictive Divergence Masks for LLM RLcited by 0
  23. 2026TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMscited by 0
  24. 2026Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?cited by 0