mlsanityData to 14 Oct 2026

Trust-Region Diffusion Policies for Massively Parallel On-Policy RL

arXiv 2606.15260 ↗First posted 13 Jun 2026cs.LG cs.AIcited by 2

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Reinforcement learning with massively parallel simulations has become a standard framework for developing robust, deployable policies; however, most existing approaches still rely on simple Gaussian policy parameterizations. Diffusion models provide a more expressive policy class and have shown strong performance on challenging control problems, yet most diffusion-based RL methods are designed for offline or off-policy training. In this work, we ask whether diffusion policies can be trained effectively in the massively parallel, on-policy regime. To this end, we introduce Trust-region Diffusion Policies (TruDi), which enables diffusion policies for on-policy RL with massively parallel simulations. This setting is particularly challenging because the data distribution changes quickly across updates, making stable training with complex policies difficult. TruDi addresses this by integrating a trust-region optimization rule to enforce a KL-divergence constraint over the entire diffusion trajectory. Empirically, we evaluate TruDi on a diverse set of 4 massively parallel RL benchmarks comprising a total of 73 tasks. Across these tasks, TruDi consistently outperforms or is on-par with strong baselines on standard tasks and achieves clear gains on more challenging humanoid control tasks, establishing a strong new baseline for massively parallel on-policy RL.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 53 of its 72 references that are in the map. First posted on arXiv 13 Jun 2026.

Builds on

53 of 72 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces26 Sep
  2. 2025Flow Matching Policy Gradients28 Jul
  3. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning28 May
  4. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learning4 Feb
  5. 2024Enhancing Exploration with Diffusion Policies in Hybrid Off-Policy RL: Application to Non-Prehensile Manipulation22 Nov
  6. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning29 Jul
  7. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient2 Jun
  8. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  9. 2024Diffusion Actor-Critic with Entropy Regulator24 May
  10. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  11. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  12. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
3 more
  1. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2016Layer Normalizationcited by 5,669
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  10. 2015Trust Region Policy Optimizationcited by 2,580
  11. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  12. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
26 more
  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  3. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  4. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  5. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  6. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  7. 2022Understanding Diffusion Models: A Unified Perspectivecited by 340
  8. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  9. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  10. 2023Denoising Diffusion Samplerscited by 149
  11. 2022An optimal control perspective on diffusion-based generative modelingcited by 147
  12. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  13. 2023Improved sampling via learned diffusionscited by 104
  14. 2018Improving Regression Performance with Distributional Lossescited by 67
  15. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrationscited by 53
  16. 2025RSL-RL: A Learning Library for Robotics Researchcited by 49
  17. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 44
  18. 2025Flow-Based Policy for Online Reinforcement Learningcited by 37
  19. 2024Simple Policy Optimizationcited by 32
  20. 2021Differentiable Trust Region Layers for Deep Reinforcement Learningcited by 25
  21. 2025VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulationcited by 25
  22. 2025Learning Sim-to-Real Humanoid Locomotion in 15 Minutescited by 16
  23. 2024Variational Distillation of Diffusion Policies into Mixture of Expertscited by 15
  24. 2024Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learningcited by 12
  25. 2025Geometry-aware RL for Manipulation of Varying Shapes and Deformable Objectscited by 10
  26. 2025Improving Long-Range Interactions in Graph Neural Simulators via Hamiltonian Dynamicscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Reachability-Aware Diffusion Policy Optimizationcited by 0
  2. 2026Reinforcement Learning via Fixed-point Flow Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning
  2. 2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Control
  3. 2024Diffusion Policy Policy Optimization
  4. 2026Diffusion Policy through Conditional Proximal Policy Optimization
  5. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learning
  6. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
  7. 2025Efficient Online Reinforcement Learning for Diffusion Policy
  8. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies