mlsanityData to 14 Oct 2026

Relative Entropy Pathwise Policy Optimization

arXiv 2507.11019 ↗First posted 15 Jul 2025cs.LGcited by 7

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Score-function based methods for policy learning, such as REINFORCE and PPO, have delivered strong results in game-playing and robotics, yet their high variance often undermines training stability. Using pathwise policy gradients, i.e. computing a derivative by differentiating the objective function, alleviates the variance issues. However, they require an accurate action-conditioned value function, which is notoriously hard to learn without relying on replay buffers for reusing past off-policy data. We present an on-policy algorithm that trains Q-value models purely from on-policy trajectories, unlocking the possibility of using pathwise policy updates in the context of on-policy learning. We show how to combine stochastic policies for exploration with constrained updates for stable training, and evaluate important architectural components that stabilize value function learning. The result, Relative Entropy Pathwise Policy Optimization (REPPO), is an efficient on-policy algorithm that combines the stability of pathwise policy gradients with the simplicity and minimal memory footprint of standard on-policy learning. Compared to state-of-the-art on two standard GPU-parallelized benchmarks, REPPO provides strong empirical performance at superior sample efficiency, wall-clock time, memory footprint, and hyperparameter robustness.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 81 of its 114 references that are in the map. First posted on arXiv 15 Jul 2025.

Builds on

81 of 114 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation28 May
  2. 2024Policy Gradient with Active Importance Sampling9 May
  3. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning1 Mar
  4. 2024Simple Policy Optimization29 Jan

Other references · 77

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2016Layer Normalizationcited by 5,669
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2015Continuous control with deep reinforcement learningcited by 3,407
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  10. 2015Trust Region Policy Optimizationcited by 2,580
  11. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
38 more
  1. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  2. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  4. 2020Mastering Atari with Discrete World Modelscited by 1,007
  5. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  8. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  9. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  10. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  11. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  12. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  13. 2018Maximum a Posteriori Policy Optimisationcited by 328
  14. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  15. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  16. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  17. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  18. 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
  19. 2020Revisiting Fundamentals of Experience Replaycited by 175
  20. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  21. 2023Understanding plasticity in neural networkscited by 174
  22. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  23. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  24. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  25. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  26. 2020Phasic Policy Gradientcited by 123
  27. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  28. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  29. 2020Mirror Descent Policy Optimizationcited by 106
  30. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  31. 2022Discovered Policy Optimisationcited by 104
  32. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  33. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89
  34. 2021Gradients are Not All You Needcited by 86
  35. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 86
  36. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  37. 2024Simplifying Deep Temporal Difference Learningcited by 78
  38. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  2. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
  3. 2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learningcited by 3
  4. 2026R^3: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learningcited by 2
  5. 2026Non-Myopic Active Feature Acquisition via Pathwise Policy Gradientscited by 1
  6. 2026FERPO: Forward Entropy-Regularized Policy Optimizationcited by 0
  7. 2026Reinforcement Learning via Fixed-point Flow Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Games
  2. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization
  3. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
  4. 2025RePO: Replay-Enhanced Policy Optimization
  5. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  6. 2026Optimistic Policy Regularization
  7. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimization
  8. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices