# A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping

arXiv: https://arxiv.org/abs/2610.10273 · HTML: https://mlsanity.com/paper/2610.10273/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://mlsanity.com/paper/2505.24298/) (2025)
- [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://mlsanity.com/paper/2005.12729/) (2020)
- [Teaching Large Language Models to Reason with Reinforcement Learning](https://mlsanity.com/paper/2403.04642/) (2024)
- [ODIN: Disentangled Reward Mitigates Hacking in RLHF](https://mlsanity.com/paper/2402.07319/) (2024)
- [Reward Shaping to Mitigate Reward Hacking in RLHF](https://mlsanity.com/paper/2502.18770/) (2025)
- [Massively Parallel Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1507.04296/) (2015)
- [Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy](https://mlsanity.com/paper/1906.10306/) (2019)
- [Improving Generalization of Alignment with Human Preferences through Group Invariant Learning](https://mlsanity.com/paper/2310.11971/) (2023)
- [CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning](https://mlsanity.com/paper/2509.20712/) (2025)
- [You May Not Need Ratio Clipping in PPO](https://mlsanity.com/paper/2202.00079/) (2022)
- [The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking](https://mlsanity.com/paper/2501.19358/) (2025)
- [It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RL](https://mlsanity.com/paper/2509.21282/) (2025)
- [Neural PPO-Clip Attains Global Optimality: A Hinge Loss Perspective](https://mlsanity.com/paper/2110.13799/) (2021)
- [An Approximate Ascent Approach To Prove Convergence of PPO](https://mlsanity.com/paper/2602.03386/) (2026)
- [Clipped-Objective Policy Gradients for Pessimistic Policy Optimization](https://mlsanity.com/paper/2311.05846/) (2023)
- [Non-Asymptotic Global Convergence of PPO-Clip](https://mlsanity.com/paper/2512.16565/) (2025)
- [CIM-PPO:Proximal Policy Optimization with Liu-Correntropy Induced Metric](https://mlsanity.com/paper/2110.10522/) (2021)
- [Sampling Complexity of TD and PPO in RKHS](https://mlsanity.com/paper/2509.24991/) (2025)

## Abstract (arXiv)

Despite its widespread use, Proximal Policy Optimization with clipping (PPO-Clip) remains difficult to tune, and the interactions among critic learning, clipping, and rollout reuse remain incompletely understood. We develop a non-asymptotic analysis of PPO-Clip as a closed-loop actor--critic system. It captures actor--critic coupling, nonsmooth probability-ratio clipping, finite-batch reuse, and predictable early stopping under explicit coverage and critic regularity assumptions, using raw GAE and Monte Carlo critic targets. Our synchronous and asynchronous guarantees jointly characterize policy stationarity and the tracking accuracy of the learned critic, with explicit dependence on algorithmic parameters. A sufficient coupling condition gives optimization, critic tracking, clipping, and finite-batch errors a common amplification bound. The asynchronous result also requires a delay-dependent critic stepsize restriction; violating these conditions does not establish divergence. For finite layered MDPs with tabular critics, a uniform bound on the actual clipped-gradient class replaces complete-trajectory counting. A verified growing-horizon family has polynomial sample complexity, and a two-time-scale schedule gives O(T-2/5) stationarity and critic-tracking bounds with explicit fresh-rollout accounting. These results together advance our understanding about PPO and provide theoretical guidance in tuning.