mlsanityData to 14 Oct 2026

Accelerating Visual-Policy Learning through Parallel Differentiable Simulation

arXiv 2505.10646 ↗First posted 15 May 2025cs.LG cs.ROcited by 5

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

In this work, we propose a computationally efficient algorithm for visual policy learning that leverages differentiable simulation and first-order analytical policy gradients. Our approach decouple the rendering process from the computation graph, enabling seamless integration with existing differentiable simulation ecosystems without the need for specialized differentiable rendering software. This decoupling not only reduces computational and memory overhead but also effectively attenuates the policy gradient norm, leading to more stable and smoother optimization. We evaluate our method on standard visual control benchmarks using modern GPU-accelerated simulation. Experiments show that our approach significantly reduces wall-clock training time and consistently outperforms all baseline methods in terms of final returns. Notably, on complex tasks such as humanoid locomotion, our method achieves a 4× improvement in final return, and successfully learns a humanoid running policy within 4 hours on a single GPU.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 43 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

32 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?18 Dec
  2. 2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation16 Dec
  3. 2024Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation4 Oct

Other references · 29

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  5. 2023Mastering Diverse Domains through World Modelscited by 1,253
  6. 2016End to End Learning for Self-Driving Carscited by 1,009
  7. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 784
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  10. 2020Reinforcement Learning with Augmented Datacited by 468
  11. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  12. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
17 more
  1. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  2. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  3. 2018Learning to Drive in a Daycited by 273
  4. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  5. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  6. 2021Learning High-Speed Flight in the Wildcited by 128
  7. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  8. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 101
  9. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89
  10. 2021Gradients are Not All You Needcited by 86
  11. 2018Simple random search provides a competitive approach to reinforcement learningcited by 75
  12. 2022Predictive Sampling: Real-time Behaviour Synthesis with MuJoCocited by 47
  13. 2021Efficient Differentiable Simulation of Articulated Bodiescited by 39
  14. 2024Learning Quadruped Locomotion Using Differentiable Simulationcited by 32
  15. 2022Training Efficient Controllers via Analytic Policy Gradientcited by 22
  16. 2024Differentiable Robot Renderingcited by 21
  17. 2024Learning Quadrotor Control From Visual Features Using Differentiable Simulationcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Reparameterization Proximal Policy Optimizationcited by 1
  2. 2026Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradientcited by 1
  3. 2026Accelerating Visual Policy Learning with Sampling-Based Model Predictive Controlcited by 0
  4. 2026Learning Vision-Based Agile Gap Traversal: Differentiable Simulation with a Warm-Started Criticcited by 0
  5. 2026Brenier Meets Adversarial Training: Optimal Transport Geometry for Robust Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer
  2. 2026SIMPLE: Simulation-Based Policy Learning and Evaluation for Humanoid Loco-manipulation
  3. 2025Learning Sim-to-Real Humanoid Locomotion in 15 Minutes
  4. 2026FetchMan: Learning Visual Humanoid Loco-Manipulation Policies from Simulated Experiences
  5. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation
  6. 2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Robotics
  7. 2025Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning
  8. 2026Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation