mlsanityData to 14 Oct 2026

Backpropagating Through Simulation: Analytic Policy Gradients for Sample and Learning Efficient Differentiable Continuous Control

arXiv 2606.21525 ↗First posted 19 Jun 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

Model-free reinforcement learning algorithms such as Proximal Policy Optimization (PPO) treat the environment as a black box, estimating policy gradients from sampled rewards; this process demands millions of interactions and relies on high-variance advantage estimates. When environment dynamics are differentiable, the return is an end-to-end differentiable function of the policy parameters, enabling exact gradient computation via backpropagation through simulation. We term this approach Analytic Policy Gradients (APG) and evaluate it against PPO on four continuous control tasks of increasing dynamical complexity: a one-dimensional point-mass target-reaching task, a 2D point-mass navigation task with obstacle avoidance, a 2D rigid-body T-block pushing task, and a 7-DOF Franka FR3 end-effector reaching task. Both algorithms share identical model architectures, observation normalization, and optimizer settings. To decouple sample efficiency from compute efficiency, we design a multi-axis evaluation protocol that records performance against environment steps and gradient steps. We report a segmented backpropagation scheme with MC and critic-based bootstrap modes that mitigates gradient degradation on long-horizon tasks, and present ablations over segment length and bootstrap strategy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 11 of its 20 references that are in the map. First posted on arXiv 19 Jun 2026.

Builds on

11 of 20 references.

References · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2023Mastering Diverse Domains through World Modelscited by 1,253
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  8. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 258
  9. 2016A Differentiable Physics Engine for Deep Learning in Roboticscited by 108
  10. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  11. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization
  2. 2026Does "Do Differentiable Simulators Give Better Policy Gradients?'' Give Better Policy Gradients?
  3. 2026Reparameterization Flow Policy Optimization
  4. 2024Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation
  5. 2024Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation
  6. 2025First Order Model-Based RL through Decoupled Backpropagation
  7. 2026Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients
  8. 2024Autonomous Vehicle Controllers From End-to-End Differentiable Simulation