mlsanityData to 14 Oct 2026

Reparameterization Proximal Policy Optimization

arXiv 2508.06214 ↗First posted 8 Aug 2025cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

By leveraging differentiable dynamics, Reparameterization Policy Gradient (RPG) achieves high sample efficiency. However, current approaches are hindered by two critical limitations: the under-utilization of computationally expensive dynamics Jacobians and inherent training instability. While sample reuse offers a remedy for under-utilization, no prior principled framework exists, and naive attempts risk exacerbating instability. To address these challenges, we propose Reparameterization Proximal Policy Optimization (RPO). We first establish that under sample reuse, RPG naturally optimizes a PPO-style surrogate objective via Backpropagation Through Time, providing a unified framework for both on- and off-policy updates. To further ensure stability, RPO integrates a clipped policy gradient mechanism tailored for RPG and employs explicit Kullback-Leibler divergence regularization. Experimental results demonstrate that RPO maintains superior sample efficiency and consistently outperforms or achieves state-of-the-art performance across diverse tasks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 42 references that are in the map. First posted on arXiv 8 Aug 2025.

Builds on

26 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Differentiable Information Enhanced Model-Based Reinforcement Learning3 Mar
  2. 2024Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation28 May

Other references · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  6. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  7. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  8. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 258
  9. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  10. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  11. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89
  12. 2021Gradients are Not All You Needcited by 86
12 more
  1. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  2. 2022Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasi-dynamic Contact Modelscited by 49
  3. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 47
  4. 2024Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physicscited by 45
  5. 2024Learning Quadruped Locomotion Using Differentiable Simulationcited by 32
  6. 2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 20
  7. 2024Do Transformer World Models Give Better Policy Gradients?cited by 10
  8. 2023Gradient Informed Proximal Policy Optimizationcited by 8
  9. 2019Total stochastic gradient algorithms and applications in reinforcement learningcited by 7
  10. 2023Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithmscited by 7
  11. 2023Improving Gradient Computation for Differentiable Physics Simulation with Contactscited by 5
  12. 2025Accelerating Visual-Policy Learning through Parallel Differentiable Simulationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026GradMAP: Gradient-Based Multi-Agent Proximal Learning for Grid-Edge Flexibilitycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Reparameterization Flow Policy Optimization
  2. 2025Sampling Complexity of TD and PPO in RKHS
  3. 2026Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?
  4. 2026KLip-PPO: A per-sample KL perspective on PPO-Clip
  5. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimization
  6. 2025Relative Entropy Pathwise Policy Optimization
  7. 2026Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence
  8. 2026ANO: Robust Policy Optimization via Bounded, Redescending Gain Fields