# Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradient

arXiv: https://arxiv.org/abs/2605.26478 · HTML: https://mlsanity.com/paper/2605.26478/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 41 of its 54 references that are in the map.
- First posted on arXiv 26 May 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [Steering Your Diffusion Policy with Latent Space Reinforcement Learning](https://mlsanity.com/paper/2506.15799/) (2025)
- [Continuous Deep Q-Learning with Model-based Acceleration](https://mlsanity.com/paper/1603.00748/) (2016)
- [Legged Locomotion in Challenging Terrains using Egocentric Vision](https://mlsanity.com/paper/2211.07638/) (2022)
- [Policy Representation via Diffusion Probability Model for Reinforcement Learning](https://mlsanity.com/paper/2305.13122/) (2023)
- [Accelerated Policy Learning with Parallel Differentiable Simulation](https://mlsanity.com/paper/2204.07137/) (2022)
- [Do Differentiable Simulators Give Better Policy Gradients?](https://mlsanity.com/paper/2202.00817/) (2022)
- [Simple random search provides a competitive approach to reinforcement learning](https://mlsanity.com/paper/1803.07055/) (2018)
- [RSL-RL: A Learning Library for Robotics Research](https://mlsanity.com/paper/2509.10771/) (2025)
- [Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo](https://mlsanity.com/paper/2212.00541/) (2022)
- [Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physics](https://mlsanity.com/paper/2407.10648/) (2024)

## Built on by (most cited first, 1 of 1)

- [Accelerating Visual Policy Learning with Sampling-Based Model Predictive Control](https://mlsanity.com/paper/2609.20575/) (2026)

## Abstract (arXiv)

We present the stochastic decoupled policy gradient (SDPG), a lightweight visual reinforcement learning (RL) method that trains diverse visuomotor control policies end-to-end within a few hours on a single NVIDIA RTX 4080 GPU. SDPG estimates policy gradients via random perturbations of trajectory rollouts, requiring orders of magnitude fewer batch-rendered environments and substantially reducing compute and memory overhead. On visual MuJoCo benchmarks, SDPG consistently outperforms baseline methods in training time, memory usage, and rewards. Finally, to support future research, we introduce a suite of realistic visual robotics benchmarks spanning dexterous manipulation, challenging locomotion, and demonstrate effective sim-to-real transfer on physical hardware.