mlsanityData to 14 Oct 2026

PPO in the Fisher-Rao geometry

arXiv 2506.03757 ↗First posted 4 Jun 2025cs.LG math.OCcited by 4

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Proximal Policy Optimization (PPO) is widely used in reinforcement learning due to its strong empirical performance, yet it lacks formal guarantees for policy improvement and convergence. PPO's clipped surrogate objective is motivated by a lower bound on linearization of the value function in flat geometry setting. We derive a tighter surrogate objective and introduce Fisher-Rao PPO (FR-PPO) by leveraging the Fisher-Rao (FR) geometry. Our scheme provides strong theoretical guarantees, including monotonic policy improvement. In the direct parametrization setting, we show that FR-PPO achieves sub-linear convergence with no dependence on action or state space dimensions, and for parametrized policies we further obtain sub-linear convergence up to the compatible function approximation error. Finally, although our primary focus is theoretical, we also demonstrate empirically that FR-PPO performs well across a range of standard reinforcement learning tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 53 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

26 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Simple Policy Optimization29 Jan

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  6. 2017Constrained Policy Optimizationcited by 983
  7. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  9. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  10. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  11. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 165
  12. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 161
13 more
  1. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  2. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 84
  3. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  4. 2019Truly Proximal Policy Optimizationcited by 69
  5. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 40
  6. 2022Mirror Descent with Relative Smoothness in Measure Spaces, with application to Sinkhorn and EMcited by 34
  7. 2023A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processescited by 29
  8. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 23
  9. 2023PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clippingcited by 17
  10. 2018A Closer Look at Deep Policy Gradientscited by 12
  11. 2022Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratiocited by 5
  12. 2024Mirror Descent for Stochastic Control Problems with Measure-valued Controlscited by 4
  13. 2013On the Performance Bounds of some Policy Search Dynamic Programming Algorithmscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Operator-Theoretic Foundations and Policy Gradient Methods for General MDPs with Unbounded Costscited by 1
  2. 2026Plasticity-Enhanced Multi-Agent Mixture of Experts for Dynamic Objective Adaptation in UAVs-Assisted Emergency Communication Networkscited by 1
  3. 2025Escaping Local Optima in the Waddington Landscape: A Two-Stage TRPO-PPO Approach for Single-Cell Perturbation Analysiscited by 0
  4. 2026PRIME: Plasticity Recovery in Multi-Agent Environments for UAV-Assisted Emergency Communication Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KLip-PPO: A per-sample KL perspective on PPO-Clip
  2. 2024Beyond the Boundaries of Proximal Policy Optimization
  3. 2023A dynamical clipping approach with task feedback for Proximal Policy Optimization
  4. 2026An Approximate Ascent Approach To Prove Convergence of PPO
  5. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization
  6. 2025Enhancing PPO with Trajectory-Aware Hybrid Policies
  7. 2026ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithm
  8. 2026ANO: Robust Policy Optimization via Bounded, Redescending Gain Fields