mlsanityData to 7 Oct 2026

Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy

arXiv 1906.10306 ↗First posted 25 Jun 2019cs.LG math.OC stat.MLcited by 42

Not on the map. Why?

Abstract (arXiv)

Proximal policy optimization and trust region policy optimization (PPO and TRPO) with actor and critic parametrized by neural networks achieve significant empirical success in deep reinforcement learning. However, due to nonconvexity, the global convergence of PPO and TRPO remains less understood, which separates theory from practice. In this paper, we prove that a variant of PPO and TRPO equipped with overparametrized neural networks converges to the globally optimal policy at a sublinear rate. The key to our analysis is the global convergence of infinite-dimensional mirror descent under a notion of one-point monotonicity, where the gradient and iterate are instantiated by neural networks. In particular, the desirable representation power and optimization geometry induced by the overparametrization of such neural networks allow them to accurately approximate the infinite-dimensional gradient and iterate.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 20; 2022: 8; 2023: 4; 2024: 5; 2025: 2; 2026: 1.

42 papers.

Sort:

By area:

  1. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  2. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  3. 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
  4. 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
  5. 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 46
  6. 2021Towards General Function Approximation in Zero-Sum Markov Gamescited by 42
  7. 2021Finite Sample Analysis of Two-Time-Scale Natural Actor-Critic Algorithmcited by 39
  8. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  9. 2023Policy Optimization for Continuous Reinforcement Learningcited by 30
  10. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithmcited by 26
  11. 2021Differentially Private Regret Minimization in Episodic Markov Decision Processescited by 23
  12. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
30 more
  1. 2022Differentially Private Reinforcement Learning with Linear Function Approximationcited by 22
  2. 2022Policy Optimization over General State and Action Spacescited by 21
  3. 2021Faster Algorithm and Sharper Analysis for Constrained Markov Decision Processcited by 18
  4. 2022Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithmcited by 17
  5. 2021Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximationcited by 15
  6. 2021Bregman Gradient Policy Optimizationcited by 15
  7. 2021Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approachcited by 13
  8. 2023Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RLcited by 13
  9. 2020When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergencecited by 12
  10. 2021Going Beyond Linear RL: Sample Efficient Neural Function Approximationcited by 9
  11. 2022Provably Efficient Fictitious Play Policy Optimization for Zero-Sum Markov Games with Structured Transitionscited by 9
  12. 2023Provably Robust Temporal Difference Learning for Heavy-Tailed Rewardscited by 9
  13. 2024Recurrent Natural Policy Gradient for POMDPscited by 8
  14. 2020Generative Adversarial Imitation Learning with Neural Networks: Global Optimality and Convergence Ratecited by 7
  15. 2021Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settingscited by 5
  16. 2021Wasserstein Flow Meets Replicator Dynamics: A Mean-Field Analysis of Representation Learning in Actor-Criticcited by 5
  17. 2022Towards Global Optimality in Cooperative MARL with the Transformation And Distillation Frameworkcited by 5
  18. 2025Improving Value Estimation Critically Enhances Vanilla Policy Gradientcited by 5
  19. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementationscited by 4
  20. 2022An Approximate Policy Iteration Viewpoint of Actor-Critic Algorithmscited by 3
  21. 2025On the Sample Complexity of Differentially Private Policy Optimizationcited by 3
  22. 2021A Stochastic Composite Augmented Lagrangian Method For Reinforcement Learningcited by 2
  23. 2021Analysis and Optimisation of Bellman Residual Errors with Neural Function Approximationcited by 2
  24. 2024Fast Convergence of Softmax Policy Mirror Ascentcited by 2
  25. 2021On the Privacy Risks of Deploying Recurrent Neural Networks in Machine Learning Modelscited by 1
  26. 2022Recursive Least Squares Advantage Actor-Critic Algorithmscited by 1
  27. 2024On the Stochastic (Variance-Reduced) Proximal Gradient Method for Regularized Expected Reward Optimizationcited by 1
  28. 2024Dual Approximation Policy Optimizationcited by 1
  29. 2023A Single-Loop Deep Actor-Critic Algorithm for Constrained Reinforcement Learning with Provable Convergencecited by 0
  30. 2026A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clippingcited by 0