mlsanityData to 14 Oct 2026

Adversarial Diffusion for Robust Reinforcement Learning

arXiv 2509.23846 ↗First posted 28 Sep 2025cs.LG cs.AIcited by 4

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Robustness to modeling errors and uncertainties remains a central challenge in reinforcement learning (RL). In this work, we address this challenge by leveraging diffusion models to train robust RL policies. Diffusion models have recently gained popularity in model-based RL due to their ability to generate full trajectories "all at once", mitigating the compounding errors typical of step-by-step transition models. Moreover, they can be conditioned to sample from specific distributions, making them highly flexible. We leverage conditional sampling to learn policies that are robust to uncertainty in environment dynamics. Building on the established connection between Conditional Value at Risk (CVaR) optimization and robust RL, we introduce Adversarial Diffusion for Robust Reinforcement Learning (AD-RRL). AD-RRL guides the diffusion process to generate worst-case trajectories during training, effectively optimizing the CVaR of the cumulative return. Empirical results across standard benchmarks show that AD-RRL achieves superior robustness and performance compared to existing robust RL methods.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 45 of its 61 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

45 of 61 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023World Models via Policy-Guided Trajectory Diffusion13 Dec
  2. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learning4 Jul
  3. 2023Value function estimation using conditional diffusion models for control9 Jun
  4. 2023MADiff: Offline Multi-agent Learning with Diffusion Models27 May
  5. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  7. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  8. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  9. 2023Mastering Diverse Domains through World Modelscited by 1,253
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  11. 2020Mastering Atari with Discrete World Modelscited by 1,007
  12. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
25 more
  1. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  2. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  3. 2019Model-Based Reinforcement Learning for Ataricited by 502
  4. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  5. 2017Robust Adversarial Reinforcement Learningcited by 451
  6. 2022Transformers are Sample-Efficient World Modelscited by 328
  7. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  8. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  9. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  10. 2019Active Domain Randomizationcited by 170
  11. 2019Benchmarking Model-Based Reinforcement Learningcited by 162
  12. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 139
  13. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 138
  14. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 101
  15. 2024Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Controlcited by 99
  16. 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 94
  17. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  18. 2016Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Modelcited by 54
  19. 2019Combating the Compounding-Error Problem with a Multi-step Modelcited by 54
  20. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 47
  21. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 41
  22. 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 28
  23. 2023A Generalist Dynamics Model for Controlcited by 24
  24. 2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 20
  25. 2022Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecificationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Efficient Tail-Aware Generative Optimization via Flow Model Fine-Tuningcited by 8
  2. 2026Receding-Horizon Control via Drifting Modelscited by 1
  3. 2026Advantage-Guided Diffusion for Model-Based Reinforcement Learningcited by 1
  4. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Robust Adversarial Policy Optimization Under Dynamics Uncertainty
  2. 2026Post-Hoc Robustness for Model-Based Reinforcement Learning
  3. 2024Safe and Robust Reinforcement Learning: Principles and Practice
  4. 2024RRLS : Robust Reinforcement Learning Suite
  5. 2024Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Model
  6. 2026Robust Policy Optimization via Adversarial Importance Sampling
  7. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms
  8. 2026Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization