# Trust Region-Guided Proximal Policy Optimization

arXiv: https://arxiv.org/abs/1901.10314 · HTML: https://mlsanity.com/paper/1901.10314/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2019.

## Built on by (most cited first, 30 of 36)

- [Generalized Proximal Policy Optimization with Sample Reuse](https://mlsanity.com/paper/2111.00072/) (2021)
- [Rethinking the Trust Region in LLM Reinforcement Learning](https://mlsanity.com/paper/2602.04879/) (2026)
- [Differentiable Trust Region Layers for Deep Reinforcement Learning](https://mlsanity.com/paper/2101.09207/) (2021)
- [GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control](https://mlsanity.com/paper/2508.03772/) (2025)
- [The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measure](https://mlsanity.com/paper/2205.10047/) (2022)
- [Spatiotemporal Costmap Inference for MPC via Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/2201.06539/) (2022)
- [Faster Deep Reinforcement Learning with Slower Online Network](https://mlsanity.com/paper/2112.05848/) (2021)
- [Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank](https://mlsanity.com/paper/2407.19943/) (2024)
- [TROLL: Trust Regions improve Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2510.03817/) (2025)
- [Decaying Clipping Range in Proximal Policy Optimization](https://mlsanity.com/paper/2102.10456/) (2021)
- [Rethinking the Divergence Regularization in LLM RL](https://mlsanity.com/paper/2606.09821/) (2026)
- [Learning to Constrain Policy Optimization with Virtual Trust Region](https://mlsanity.com/paper/2204.09315/) (2022)
- [Safe and Robust Reinforcement Learning: Principles and Practice](https://mlsanity.com/paper/2403.18539/) (2024)
- [BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning](https://mlsanity.com/paper/2603.04918/) (2026)
- [Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning](https://mlsanity.com/paper/2606.10968/) (2026)
- [Overcoming Non-stationary Dynamics with Evidential Proximal Policy Optimization](https://mlsanity.com/paper/2503.01468/) (2025)
- [Curious Explorer: a provable exploration strategy in Policy Learning](https://mlsanity.com/paper/2106.15503/) (2021)
- [Generalized Policy Improvement Algorithms with Theoretically Supported Sample Reuse](https://mlsanity.com/paper/2206.13714/) (2022)
- [Distributed Stochastic Gradient Descent with Staleness: A Stochastic Delay Differential Equation Based Framework](https://mlsanity.com/paper/2406.11159/) (2024)
- [Stabilizing Policy Gradient Methods via Reward Profiling](https://mlsanity.com/paper/2511.16629/) (2025)
- [ETR: Outcome-Guided Elastic Trust Regions for Policy Optimization](https://mlsanity.com/paper/2601.03723/) (2026)
- [Plasticity-Enhanced Multi-Agent Mixture of Experts for Dynamic Objective Adaptation in UAVs-Assisted Emergency Communication Networks](https://mlsanity.com/paper/2604.09028/) (2026)
- [Processing Network Controls via Deep Reinforcement Learning](https://mlsanity.com/paper/2205.02119/) (2022)
- [Entropy Augmented Reinforcement Learning](https://mlsanity.com/paper/2208.09322/) (2022)
- [Joint action loss for proximal policy optimization](https://mlsanity.com/paper/2301.10919/) (2023)
- [A dynamical clipping approach with task feedback for Proximal Policy Optimization](https://mlsanity.com/paper/2312.07624/) (2023)
- [Escaping Local Optima in the Waddington Landscape: A Two-Stage TRPO-PPO Approach for Single-Cell Perturbation Analysis](https://mlsanity.com/paper/2510.13018/) (2025)
- [Directional-Clamp PPO](https://mlsanity.com/paper/2511.02577/) (2025)
- [A Unified Framework for Rethinking Policy Divergence Measures in GRPO](https://mlsanity.com/paper/2602.05494/) (2026)
- [ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithm](https://mlsanity.com/paper/2602.09726/) (2026)

## Abstract (arXiv)

Proximal policy optimization (PPO) is one of the most popular deep reinforcement learning (RL) methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, as a model-free RL method, the success of PPO relies heavily on the effectiveness of its exploratory policy search. In this paper, we give an in-depth analysis on the exploration behavior of PPO, and show that PPO is prone to suffer from the risk of lack of exploration especially under the case of bad initialization, which may lead to the failure of training or being trapped in bad local optima. To address these issues, we proposed a novel policy optimization method, named Trust Region-Guided PPO (TRGPPO), which adaptively adjusts the clipping range within the trust region. We formally show that this method not only improves the exploration ability within the trust region but enjoys a better performance bound compared to the original PPO as well. Extensive experiments verify the advantage of the proposed method.