mlsanityData to 7 Oct 2026

Do Differentiable Simulators Give Better Policy Gradients?

arXiv 2202.00817 ↗First posted 2 Feb 2022cs.LG cs.AI cs.ROcited by 83

Not on the map. Why?

Abstract (arXiv)

Differentiable simulators promise faster computation time for reinforcement learning by replacing zeroth-order gradient estimates of a stochastic objective with an estimate based on first-order gradients. However, it is yet unclear what factors decide the performance of the two estimators on complex landscapes that involve long-horizon planning and control on physical systems, despite the crucial relevance of this question for the utility of differentiable simulators. We show that characteristics of certain physical systems, such as stiffness or discontinuities, may compromise the efficacy of the first-order estimator, and analyze this phenomenon through the lens of bias and variance. We additionally propose an α-order gradient estimator, with α ∈ [0,1], which correctly utilizes exact gradients to combine the efficiency of first-order estimates with the robustness of zero-order methods. We demonstrate the pitfalls of traditional estimators and the advantages of the α-order estimator on some numerical examples.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 24 of its 59 references that are in the map. First posted on arXiv 2 Feb 2022.

Builds on

24 of 59 references.

References · 24

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Convex optimizationcited by 2,479
  4. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  5. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  6. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  7. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  8. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
  9. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 248
  10. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 211
  11. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 190
  12. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
12 more
  1. 2015Gradient Estimation Using Stochastic Computation Graphscited by 120
  2. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 117
  3. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
  4. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  5. 2021Gradients are Not All You Needcited by 82
  6. 2019A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimizationcited by 66
  7. 2021Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contactcited by 50
  8. 2021Bundled Gradients through Contact via Randomized Smoothingcited by 23
  9. 2020ADD: Analytically Differentiable Dynamics for Multi-Body Systems with Frictional Contactcited by 17
  10. 2021Deluca -- A Differentiable Control Library: Environments, Methods, and Benchmarkingcited by 10
  11. 2019A pressure field model for fast, robust approximation of net contact force and moment between nominally rigid objectscited by 8
  12. 2019A Transition-Aware Method for the Simulation of Compliant Contact with Regularized Frictioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 22; 2024: 18; 2025: 16; 2026: 21.

83 papers.

Sort:

By area:

  1. 2023DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physicscited by 23
  2. 2022Deep Inventory Managementcited by 22
  3. 2024Learning Quadruped Locomotion Using Differentiable Simulationcited by 22
  4. 2022Rethinking Optimization with Differentiable Simulation from a Global Perspectivecited by 21
  5. 2023SoftZoo: A Soft Robot Co-design Benchmark For Locomotion In Diverse Environmentscited by 18
  6. 2024Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulationcited by 18
  7. 2023DiffMimic: Efficient Motion Mimicking with Differentiable Physicscited by 17
  8. 2023Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matchingcited by 17
  9. 2024PWM: Policy Learning with Multi-Task World Modelscited by 17
  10. 2023Faster Gradient-Free Algorithms for Nonsmooth Nonconvex Stochastic Optimizationcited by 16
  11. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  12. 2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 14
38 more
  1. 2024A Taxonomy of Loss Functions for Stochastic Optimal Controlcited by 11
  2. 2022Plateau-reduced Differentiable Path Tracingcited by 10
  3. 2024Do Transformer World Models Give Better Policy Gradients?cited by 10
  4. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Modelscited by 10
  5. 2022Learning Tool Morphology for Contact-Rich Manipulation Tasks with Differentiable Simulationcited by 9
  6. 2023Smoothed Online Learning for Prediction in Piecewise Affine Systemscited by 9
  7. 2023Fractal Landscapes in Policy Optimizationcited by 8
  8. 2023Gradient Informed Proximal Policy Optimizationcited by 8
  9. 2023Differentiable Simulations for Enhanced Sampling of Rare Eventscited by 7
  10. 2023Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithmscited by 7
  11. 2023Differentiable Visual Computing for Inverse Problems and Machine Learningcited by 7
  12. 2022Differentiable Physics Simulations with Contacts: Do They Have Correct Gradients w.r.t. Position, Velocity and Control?cited by 6
  13. 2023DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physicscited by 6
  14. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  15. 2022Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learningcited by 5
  16. 2023Oracle-Efficient Smoothed Online Learning for Piecewise Continuous Decision Makingcited by 5
  17. 2023Improving Gradient Computation for Differentiable Physics Simulation with Contactscited by 5
  18. 2023Zero Grads: Learning Local Surrogate Losses for Non-Differentiable Graphicscited by 5
  19. 2023Learning an Inventory Control Policy with General Inventory Arrival Dynamicscited by 5
  20. 2024Equality Constrained Diffusion for Direct Trajectory Optimizationcited by 5
  21. 2025Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?cited by 5
  22. 2024Generalizing Stochastic Smoothing for Differentiation and Gradient Estimationcited by 4
  23. 2025Accelerating Visual-Policy Learning through Parallel Differentiable Simulationcited by 4
  24. 2025Time-Aware World Model for Adaptive Prediction and Controlcited by 4
  25. 2023Generating Transferable Adversarial Simulation Scenarios for Self-Driving via Neural Renderingcited by 3
  26. 2023Robust Learning for Smoothed Online Convex Optimization with Feedback Delaycited by 3
  27. 2024Stabilizing Backpropagation Through Time to Learn Complex Physicscited by 3
  28. 2024Differentiable Discrete Event Simulation for Queuing Network Controlcited by 3
  29. 2025Efficient Inverse Multiagent Learningcited by 3
  30. 2025Differentiable Information Enhanced Model-Based Reinforcement Learningcited by 3
  31. 2025First Order Model-Based RL through Decoupled Backpropagationcited by 3
  32. 2026Robustness without Wrinkles: Parallel Simulation and Robust MPC for Certified Deformable Manipulationcited by 3
  33. 2024Mollification Effects of Policy Gradient Methodscited by 2
  34. 2024DiffSim2Real: Deploying Quadrupedal Locomotion Policies Purely Trained in Differentiable Simulationcited by 2
  35. 2024Is Linear Feedback on Smoothed Dynamics Sufficient for Stabilizing Contact-Rich Plans?cited by 2
  36. 2024Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimationcited by 2
  37. 2025On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimizationcited by 2
  38. 2023Diff-Transfer: Model-based Robotic Manipulation Skill Transfer via Differentiable Physics Simulationcited by 1

50 most cited of 83 All 83 (JSON)