mlsanityData to 7 Oct 2026

Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation

arXiv 2405.17784 ↗First posted 28 May 2024cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Model-Free Reinforcement Learning (MFRL), leveraging the policy gradient theorem, has demonstrated considerable success in continuous control tasks. However, these approaches are plagued by high gradient variance due to zeroth-order gradient estimation, resulting in suboptimal policies. Conversely, First-Order Model-Based Reinforcement Learning (FO-MBRL) methods employing differentiable simulation provide gradients with reduced variance but are susceptible to sampling error in scenarios involving stiff dynamics, such as physical contact. This paper investigates the source of this error and introduces Adaptive Horizon Actor-Critic (AHAC), an FO-MBRL algorithm that reduces gradient error by adapting the model-based horizon to avoid stiff dynamics. Empirical findings reveal that AHAC outperforms MFRL baselines, attaining 40% more reward across a set of locomotion tasks and efficiently scaling to high-dimensional control environments with improved wall-clock-time efficiency.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

33 of 46 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2023Mastering Diverse Domains through World Modelscited by 1,193
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2018DeepMind Control Suitecited by 935
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  10. 2019Solving Rubik's Cube with a Robot Handcited by 699
  11. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  12. 2019Learning agile and dynamic motor skills for legged robotscited by 402
21 more
  1. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  2. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  3. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
  4. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 248
  5. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  6. 2015Gradient Estimation Using Stochastic Computation Graphscited by 120
  7. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
  8. 2018ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Roboticscited by 98
  9. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
  10. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 83
  11. 2017Control of a Quadrotor with Reinforcement Learningcited by 75
  12. 2019A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimizationcited by 66
  13. 2020Deep Drone Acrobaticscited by 66
  14. 2021DiffPD: Differentiable Projective Dynamicscited by 62
  15. 2021DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cuttingcited by 62
  16. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  17. 2021An End-to-End Differentiable Framework for Contact-Aware Robot Designcited by 53
  18. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 46
  19. 2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
  20. 2019Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Modelscited by 26
  21. 2023Improving Gradient Computation for Differentiable Physics Simulation with Contactscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 9; 2026: 4.

18 papers.

Sort:

By area:

  1. 2025Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Roboticscited by 47
  2. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  3. 2024PWM: Policy Learning with Multi-Task World Modelscited by 17
  4. 2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 14
  5. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  6. 2025KinTwin: Imitation Learning with Torque and Muscle Driven Biomechanical Models Enables Precise Replication of Able-Bodied and Impaired Movement from Markerless Motion Capturecited by 5
  7. 2025Differentiable Simulation of Hard Contacts with Soft Gradients for Learning and Controlcited by 5
  8. 2026Reparameterization Flow Policy Optimizationcited by 4
  9. 2025First Order Model-Based RL through Decoupled Backpropagationcited by 3
  10. 2024DiffSim2Real: Deploying Quadrupedal Locomotion Policies Purely Trained in Differentiable Simulationcited by 2
  11. 2025Accelerated Learning with Linear Temporal Logic using Differentiable Simulationcited by 1
  12. 2025Leveraging Analytic Gradients in Provably Safe Reinforcement Learningcited by 1
6 more
  1. 2025Reparameterization Proximal Policy Optimizationcited by 1
  2. 2024Improving generalization of robot locomotion policies via Sharpness-Aware Reinforcement Learningcited by 0
  3. 2025Bayesian Inverse Physics for Neuro-Symbolic Robot Learningcited by 0
  4. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0
  5. 2026Open-DiffLoco: Open-Source Differentiable Learning for Deployable Blind Quadruped Locomotioncited by 0
  6. 2026When Does Backpropagating Through Policy Memory Matter? Physical Credit, Optimizer Updates, and Observabilitycited by 0