mlsanityData to 7 Oct 2026

Leveraging Reward Gradients For Reinforcement Learning in Differentiable Physics Simulations

arXiv 2203.02857 ↗First posted 6 Mar 2022cs.LG cs.RO cs.SYcited by 4

Not on the map. Why?

Abstract (arXiv)

In recent years, fully differentiable rigid body physics simulators have been developed, which can be used to simulate a wide range of robotic systems. In the context of reinforcement learning for control, these simulators theoretically allow algorithms to be applied directly to analytic gradients of the reward function. However, to date, these gradients have proved extremely challenging to use, and are outclassed by algorithms using no gradient information at all. In this work we present a novel algorithm, cross entropy analytic policy gradients, that is able to leverage these gradients to outperform state of art deep reinforcement learning on a set of challenging nonlinear control problems.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 21 of its 36 references that are in the map. First posted on arXiv 6 Mar 2022.

Builds on

21 of 36 references.

References · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2018Learning Dexterous In-Hand Manipulationcited by 643
  6. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  7. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  8. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 299
  9. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 248
  10. 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 239
  11. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 159
  12. 2016A Differentiable Physics Engine for Deep Learning in Roboticscited by 107
9 more
  1. 2021Gradients are Not All You Needcited by 82
  2. 2020NeuralSim: Augmenting Differentiable Simulators with Neural Networkscited by 77
  3. 2021Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learningcited by 73
  4. 2018CEM-RL: Combining evolutionary and gradient-based methods for policy searchcited by 53
  5. 2021Efficient Differentiable Simulation of Articulated Bodiescited by 38
  6. 2020Model-Predictive Control via Cross-Entropy and Gradient-Based Optimizationcited by 32
  7. 2019Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learningcited by 30
  8. 2021CEM-GD: Cross-Entropy Method with Gradient Descent Planner for Model-Based Reinforcement Learningcited by 6
  9. 2020Combining Deep Reinforcement Learning And Local Control For The Acrobot Swing-up And Balance Taskcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 3.

4 papers.

  1. 2022Training Efficient Controllers via Analytic Policy Gradientcited by 16
  2. 2024Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Rewardcited by 16
  3. 2024Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial Statescited by 4
  4. 2024Autonomous Vehicle Controllers From End-to-End Differentiable Simulationcited by 1