mlsanityData to 7 Oct 2026

Non-Asymptotic Global Convergence of PPO-Clip

arXiv 2512.16565 ↗First posted 18 Dec 2025math.OC cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning has gained attention for modern Large Language Model post-training. The actor-only variants of Proximal Policy Optimization (PPO) are widely applied for their efficiency. These algorithms incorporate a clipping mechanism to improve stability. Besides, a regularization term, such as the reverse KL-divergence or a more general (f)-divergence, is introduced to control excessive deviation from a reference policy. Despite their empirical success, a rigorous theoretical understanding of the problem and the algorithm's properties is limited. This paper advances the theoretical foundations of the PPO-Clip algorithm by analyzing a deterministic actor-only PPO algorithm within the general RL setting with (f)-divergence regularization under the softmax policy parameterization. We derive a non-uniform Lipschitz smoothness condition and a Lojasiewicz inequality for the considered problem. Based on these properties, we establish non-asymptotic global linear convergence in value gap for the forward KL regularizer. For the reverse KL regularizer, we derive global linear convergence from any finite softmax initialization in both value gap and squared policy distance.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

39 of 54 references.

References · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2022Red Teaming Language Models with Language Modelscited by 983
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  10. 2019A Theory of Regularized Markov Decision Processescited by 279
  11. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  12. 2018Provably Efficient Maximum Entropy Explorationcited by 240
27 more
  1. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  2. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  3. 2017A unified view of entropy-regularized Markov decision processescited by 174
  4. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  5. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  6. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  7. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
  8. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  9. 2020A Distributional Approach to Controlled Text Generationcited by 117
  10. 2022On the Convergence Rates of Policy Gradient Methodscited by 115
  11. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  12. 2022On Decoding Strategies for Neural Text Generatorscited by 89
  13. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 85
  14. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
  15. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  16. 2021A general sample complexity analysis of vanilla policy gradientcited by 74
  17. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  18. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 63
  19. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 53
  20. 2020Leverage the Average: an Analysis of KL Regularization in RLcited by 33
  21. 2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
  22. 2023Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processescited by 24
  23. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 23
  24. 2020Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDERcited by 19
  25. 2023PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clippingcited by 17
  26. 2020Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Samplingcited by 15
  27. 2022PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policynesscited by 0
  2. 2026A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clippingcited by 0