mlsanityData to 14 Oct 2026

PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping

arXiv 2312.12065 ↗First posted 19 Dec 2023cs.LG cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Proximal Policy Optimization algorithm employing a clipped surrogate objective (PPO-Clip) is a prominent exemplar of the policy optimization methods. However, despite its remarkable empirical success, PPO-Clip lacks theoretical substantiation to date. In this paper, we contribute to the field by establishing the first global convergence results of a PPO-Clip variant in both tabular and neural function approximation settings. Our findings highlight the O(1/√T) min-iterate convergence rate specifically in the context of neural function approximation. We tackle the inherent challenges in analyzing PPO-Clip through three central concepts: (i) We introduce a generalized version of the PPO-Clip objective, illuminated by its connection with the hinge loss. (ii) Employing entropic mirror descent, we establish asymptotic convergence for tabular PPO-Clip with direct policy parameterization. (iii) Inspired by the tabular analysis, we streamline convergence analysis by introducing a two-step policy improvement approach. This decouples policy search from complex neural policy parameterization using a regression-based update scheme. Furthermore, we gain deeper insights into the efficacy of PPO-Clip by interpreting these generalized objectives. Our theoretical findings also mark the first characterization of the influence of the clipping mechanism on PPO-Clip convergence. Importantly, the clipping range affects only the pre-constant of the convergence rate.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 18 of its 33 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

18 of 33 references.

References · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2016OpenAI Gymcited by 1,736
  4. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  5. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  6. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  7. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  8. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 161
  9. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  10. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  11. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  12. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
6 more
  1. 2013Accelerated Gradient Methods for Nonconvex Nonlinear and Stochastic Programmingcited by 75
  2. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 68
  3. 2016Convergence Rate of Frank-Wolfe for Non-Convex Objectivescited by 63
  4. 2020Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Timecited by 20
  5. 2018An Adaptive Clipping Approach for Proximal Policy Optimizationcited by 14
  6. 2020Proximal Policy Gradient: PPO with Policy Gradientcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 6.

17 papers.

Sort:

By area:

  1. 2025Process-Supervised Reinforcement Learning for Code Generationcited by 22
  2. 2025Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policiescited by 22
  3. 2024HM3: Hierarchical Multi-Objective Model Merging for Pretrained Modelscited by 8
  4. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Modelscited by 7
  5. 2025PPO in the Fisher-Rao geometrycited by 4
  6. 2025PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Modelscited by 4
  7. 2025Test-Time Alignment for Large Language Models via Textual Model Predictive Controlcited by 2
  8. 2025Optimizing Navigation And Chemical Application in Precision Agriculture With Deep Reinforcement Learning And Conditional Action Treecited by 2
  9. 2025Non-Asymptotic Global Convergence of PPO-Clipcited by 2
  10. 2026Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensuscited by 2
  11. 2025SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Spacecited by 1
  12. 2026ReLA: Representation Learning and Aggregation for Job Scheduling with Reinforcement Learningcited by 1
5 more
  1. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 1
  2. 2025Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spacescited by 0
  3. 2025Directional-Clamp PPOcited by 0
  4. 2026Plan2Cleanse: Test-Time Backdoor Defense via Monte-Carlo Planning in Deep Reinforcement Learningcited by 0
  5. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithmcited by 0