mlsanityData to 14 Oct 2026

Central Path Proximal Policy Optimization

arXiv 2506.00700 ↗First posted 31 May 2025cs.LGcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In constrained Markov decision processes, enforcing constraints during training is often thought of as decreasing the final return. Recently, it was shown that constraints can be incorporated directly into the policy geometry, yielding an optimization trajectory close to the central path of a barrier method, which does not compromise final return. Building on this idea, we introduce Central Path Proximal Policy Optimization (C3PO), a simple modification of the PPO loss that produces policy iterates, that stay close to the central path of the constrained optimization problem. Compared to existing on-policy methods, C3PO delivers improved performance with tighter constraint enforcement, suggesting that central path-guided updates offer a promising direction for constrained policy optimization.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 39 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

27 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Optimal Rates of Convergence for Entropy Regularization in Discounted Markov Decision Processes6 Jun

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Convex optimizationcited by 2,515
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2017Constrained Policy Optimizationcited by 983
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  8. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  9. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  10. 2019A Theory of Regularized Markov Decision Processescited by 286
  11. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  12. 2017A unified view of entropy-regularized Markov decision processescited by 182
14 more
  1. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  2. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  3. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  4. 2021Reward is enough for convex MDPscited by 84
  5. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
  6. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 31
  7. 2018Hessian Riemannian gradient flows in convex programmingcited by 26
  8. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 23
  9. 2024Constrained Reinforcement Learning with Smoothed Log Barrier Functioncited by 15
  10. 2022Geometry and convergence of natural policy gradient methodscited by 12
  11. 2024On PI Controllers for Updating Lagrange Multipliers in Constrained Optimizationcited by 12
  12. 2022Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learningcited by 11
  13. 2023A safe exploration approach to constrained Markov decision processescited by 10
  14. 2024Embedding Safety into RL: A New Take on Trust Region Methodscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025The Geometry of Nonlinear Reinforcement Learningcited by 1
  2. 2025SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraintscited by 0
  3. 2026Bounded Ratio Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing PPO with Trajectory-Aware Hybrid Policies
  2. 2025Incentivizing Safer Actions in Policy Optimization for Constrained Reinforcement Learning
  3. 2025Proactive Constrained Policy Optimization with Preemptive Penalty
  4. 2024Strongly-polynomial time and validation analysis of policy gradient methods
  5. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form
  6. 2025Learning Deterministic Policies with Policy Gradients in Constrained Markov Decision Processes
  7. 2026CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
  8. 2026KLip-PPO: A per-sample KL perspective on PPO-Clip