# An Adaptive Clipping Approach for Proximal Policy Optimization

arXiv: https://arxiv.org/abs/1804.06461 · HTML: https://mlsanity.com/paper/1804.06461/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2018.

## Built on by (most cited first, 14 of 14)

- [PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping](https://mlsanity.com/paper/2312.12065/) (2023)
- [Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models](https://mlsanity.com/paper/2510.09976/) (2025)
- [Decaying Clipping Range in Proximal Policy Optimization](https://mlsanity.com/paper/2102.10456/) (2021)
- [Neural PPO-Clip Attains Global Optimality: A Hinge Loss Perspective](https://mlsanity.com/paper/2110.13799/) (2021)
- [Autonomous Drone Swarm Navigation and Multi-target Tracking in 3D Environments with Dynamic Obstacles](https://mlsanity.com/paper/2202.06253/) (2022)
- [BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning](https://mlsanity.com/paper/2603.04918/) (2026)
- [CIM-PPO:Proximal Policy Optimization with Liu-Correntropy Induced Metric](https://mlsanity.com/paper/2110.10522/) (2021)
- [Optimizing Electric Bus Charging Scheduling with Uncertainties Using Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/2505.10296/) (2025)
- [On Multi-Agent Deep Deterministic Policy Gradients and their Explainability for SMARTS Environment](https://mlsanity.com/paper/2301.09420/) (2023)
- [A dynamical clipping approach with task feedback for Proximal Policy Optimization](https://mlsanity.com/paper/2312.07624/) (2023)
- [Curriculum Is More Influential Than Haptic Information During Reinforcement Learning of Object Manipulation Against Gravity](https://mlsanity.com/paper/2407.09986/) (2024)
- [Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spaces](https://mlsanity.com/paper/2508.14102/) (2025)
- [Safe and Sustainable Electric Bus Charging Scheduling with Constrained Hierarchical DRL](https://mlsanity.com/paper/2512.03059/) (2025)
- [Group Adaptive Clipping Policy Optimization](https://mlsanity.com/paper/2609.00444/) (2026)

## Abstract (arXiv)

Very recently proximal policy optimization (PPO) algorithms have been proposed as first-order optimization methods for effective reinforcement learning. While PPO is inspired by the same learning theory that justifies trust region policy optimization (TRPO), PPO substantially simplifies algorithm design and improves data efficiency by performing multiple epochs of clipped policy optimization from sampled data. Although clipping in PPO stands for an important new mechanism for efficient and reliable policy update, it may fail to adaptively improve learning performance in accordance with the importance of each sampled state. To address this issue, a new surrogate learning objective featuring an adaptive clipping mechanism is proposed in this paper, enabling us to develop a new algorithm, known as PPO-λ. PPO-λ optimizes policies repeatedly based on a theoretical target for adaptive policy improvement. Meanwhile, destructively large policy update can be effectively prevented through both clipping and adaptive control of a hyperparameter λ in PPO-λ, ensuring high learning reliability. PPO-λ enjoys the same simple and efficient design as PPO. Empirically on several Atari game playing tasks and benchmark control tasks, PPO-λ also achieved clearly better performance than PPO.