mlsanityData to 14 Oct 2026

Where-to-Learn: Analytical Policy Gradient Directed Exploration for On-Policy Robotic Reinforcement Learning

arXiv 2603.27317 ↗First posted 28 Mar 2026cs.ROcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

On-policy reinforcement learning (RL) algorithms have demonstrated great potential in robotic control, where effective exploration is crucial for efficient and high-quality policy learning. However, how to encourage the agent to explore the better trajectories efficiently remains a challenge. Most existing methods incentivize exploration by maximizing the policy entropy or encouraging novel state visiting regardless of the potential state value. We propose a new form of directed exploration that uses analytical policy gradients from a differentiable dynamics model to inject task-aware, physics-guided guidance, thereby steering the agent towards high-reward regions for accelerated and more effective policy learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 45 references that are in the map. First posted on arXiv 28 Mar 2026.

Builds on

26 of 45 references.

References · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2018Exploration by Random Network Distillationcited by 993
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  10. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  11. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  12. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
14 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  2. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
  3. 2017Parameter Space Noise for Explorationcited by 161
  4. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  5. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  6. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89
  7. 2021Gradients are Not All You Needcited by 86
  8. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 63
  9. 2024Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physicscited by 45
  10. 2021Efficient Differentiable Simulation of Articulated Bodiescited by 39
  11. 2024Learning Quadruped Locomotion Using Differentiable Simulationcited by 32
  12. 2024Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulationcited by 13
  13. 2024Learning Deployable Locomotion Control via Differentiable Simulationcited by 8
  14. 2022Differentiable Physics Simulations with Contacts: Do They Have Correct Gradients w.r.t. Position, Velocity and Control?cited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Physics-Informed Policy Optimization via Analytic Dynamics Regularization
  2. 2025Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Space
  3. 2024Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance
  4. 2025Leveraging Analytic Gradients in Provably Safe Reinforcement Learning
  5. 2024Diffusion Policy Policy Optimization
  6. 2026Backpropagating Through Simulation: Analytic Policy Gradients for Sample and Learning Efficient Differentiable Continuous Control
  7. 2026Policy Gradient Methods for Non-Markovian Reinforcement Learning
  8. 2025SpeedAug: Policy Acceleration via Tempo-Enriched Policy and RL Fine-Tuning