mlsanityData to 14 Oct 2026

Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation

arXiv 2412.12089 ↗First posted 16 Dec 2024cs.LG cs.AI cs.CVcited by 20

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Recent advances in GPU-based parallel simulation have enabled practitioners to collect large amounts of data and train complex control policies using deep reinforcement learning (RL), on commodity GPUs. However, such successes for RL in robotics have been limited to tasks sufficiently simulated by fast rigid-body dynamics. Simulation techniques for soft bodies are comparatively several orders of magnitude slower, thereby limiting the use of RL due to sample complexity requirements. To address this challenge, this paper presents both a novel RL algorithm and a simulation platform to enable scaling RL on tasks involving rigid bodies and deformables. We introduce Soft Analytic Policy Optimization (SAPO), a maximum entropy first-order model-based actor-critic RL algorithm, which uses first-order analytic gradients from differentiable simulation to train a stochastic actor to maximize expected return and entropy. Alongside our approach, we develop Rewarped, a parallel differentiable multiphysics simulation platform that supports simulating various materials beyond rigid bodies. We re-implement challenging manipulation and locomotion tasks in Rewarped, and show that SAPO outperforms baselines over a range of tasks that involve interaction between rigid bodies, articulations, and deformables. Additional details at https://rewarped.github.io/.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 88 of its 134 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

88 of 134 references.

References · 88

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Layer Normalizationcited by 5,669
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  10. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  11. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
38 more
  1. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  2. 2019Solving Rubik's Cube with a Robot Handcited by 734
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  4. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  5. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  6. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  7. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  8. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  9. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 295
  10. 2018Distributed Prioritized Experience Replaycited by 269
  11. 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
  12. 2018Understanding the impact of entropy on policy optimizationcited by 176
  13. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  14. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  15. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  16. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  17. 2015Gradient Estimation Using Stochastic Computation Graphscited by 121
  18. 2020SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulationcited by 121
  19. 2016A Differentiable Physics Engine for Deep Learning in Roboticscited by 108
  20. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  21. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 103
  22. 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
  23. 2015Gaussian Processes for Data-Efficient Learning in Robotics and Controlcited by 98
  24. 2018ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Roboticscited by 98
  25. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  26. 2017Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulationcited by 90
  27. 2018GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learningcited by 90
  28. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89
  29. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  30. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 82
  31. 2020NeuralSim: Augmenting Differentiable Simulators with Neural Networkscited by 82
  32. 2021FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfoldingcited by 81
  33. 2024Simplifying Deep Temporal Difference Learningcited by 78
  34. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 76
  35. 2020Scalable Differentiable Physics for Learning and Controlcited by 73
  36. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 72
  37. 2021DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contactcited by 65
  38. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 63

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 12.

20 papers.

Sort:

By area:

  1. 2025Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Taskscited by 11
  2. 2025Accelerating Visual-Policy Learning through Parallel Differentiable Simulationcited by 5
  3. 2025First Order Model-Based RL through Decoupled Backpropagationcited by 5
  4. 2026Reparameterization Flow Policy Optimizationcited by 4
  5. 2025Bridging VLMs and Embodied Intelligence with Deliberate Practice Policy Optimizationcited by 3
  6. 2026Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Controlcited by 2
  7. 2025Leveraging Analytic Gradients in Provably Safe Reinforcement Learningcited by 1
  8. 2025Reparameterization Proximal Policy Optimizationcited by 1
  9. 2026Coupled Local and Global World Models for Efficient First Order RLcited by 1
  10. 2026Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradientcited by 1
  11. 2025Physics-Informed Reinforcement Learning for Large-Scale EV Smart Charging Considering Distribution Network Voltage Constraintscited by 0
  12. 2025First-order Sobolev Reinforcement Learningcited by 0
8 more
  1. 2026VisFly-Lab: Unified Differentiable Framework for First-Order Reinforcement Learning of Quadrotor Controlcited by 0
  2. 2026Does "Do Differentiable Simulators Give Better Policy Gradients?'' Give Better Policy Gradients?cited by 0
  3. 2026SILO: Simulation-in-the-Loop Sim-to-Real Transfer for Multi-Stage Cable Routingcited by 0
  4. 2026Size Doesn't Matter: Material-State Reinforcement Learning for Excavator Transferable Soil Manipulationcited by 0
  5. 2026Accelerating Visual Policy Learning with Sampling-Based Model Predictive Controlcited by 0
  6. 2026When Does Backpropagating Through Policy Memory Matter? Physical Credit, Optimizer Updates, and Observabilitycited by 0
  7. 2026ClearNav: Perception-Aligned Quadrotor Flight Using Differential-Physicscited by 0
  8. 2026E2DRL: End-to-End Differentiable Reinforcement Learning for 3D Obstacle Avoidance in Quadruped Robotscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Physics-Informed Policy Optimization via Analytic Dynamics Regularization
  2. 2026Batched Differentiable Rigid Body Dynamics in PyTorch for GPU-Accelerated Robot Learning
  3. 2024Improving generalization of robot locomotion policies via Sharpness-Aware Reinforcement Learning
  4. 2026Kamino: GPU-based Massively Parallel Simulation of Multi-Body Systems with Challenging Topologies ↗
  5. 2024Real-World Fluid Directed Rigid Body Control via Deep Reinforcement Learning
  6. 2024Highly-Efficient Differentiable Simulation for Robotics
  7. 2026Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control
  8. 2026Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation