mlsanityData to 7 Oct 2026

D2 Actor Critic: Diffusion Actor Meets Distributional Critic

arXiv 2510.03508 ↗First posted 3 Oct 2025cs.LGcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

We introduce D2AC, a new model-free reinforcement learning (RL) algorithm designed to train expressive diffusion policies online effectively. At its core is a policy improvement objective that avoids the high variance of typical policy gradients and the complexity of backpropagation through time. This stable learning process is critically enabled by our second contribution: a robust distributional critic, which we design through a fusion of distributional RL and clipped double Q-learning. The resulting algorithm is highly effective, achieving state-of-the-art performance on a benchmark of eighteen hard RL tasks, including Humanoid, Dog, and Shadow Hand domains, spanning both dense-reward and goal-conditioned RL scenarios. Beyond standard benchmarks, we also evaluate a biologically motivated predator-prey task to examine the behavioral robustness and generalization capacity of our approach. Code: https://github.com/d2ac-actor-critic/d2ac-public

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 56 of its 69 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

56 of 69 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Distributional Soft Actor-Critic with Diffusion Policy2 Jul
  2. 2025Maximum Entropy Reinforcement Learning with Diffusion Policy17 Feb
  3. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning7 Feb
  4. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  5. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  6. 2023Boosting Continuous Control with Consistency Policy10 Oct
  7. 2023Training Diffusion Models with Reinforcement Learning22 May
  8. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  9. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2016Layer Normalizationcited by 5,579
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2015Continuous control with deep reinforcement learningcited by 3,279
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  12. 2015Trust Region Policy Optimizationcited by 2,503
34 more
  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2023Consistency Modelscited by 1,778
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2023Mastering Diverse Domains through World Modelscited by 1,193
  6. 2021Variational Diffusion Modelscited by 1,145
  7. 2017Hindsight Experience Replaycited by 1,096
  8. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  9. 2018DeepMind Control Suitecited by 935
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  12. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  13. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  14. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  15. 2023Improved Techniques for Training Consistency Modelscited by 430
  16. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  17. 2018Maximum a Posteriori Policy Optimisationcited by 318
  18. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  19. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  20. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  21. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  22. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  23. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  24. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  25. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  26. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  27. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  28. 2018Improving Regression Performance with Distributional Lossescited by 66
  29. 2024Flash Diffusion: Accelerating Any Conditional Diffusion Model for Few Steps Image Generationcited by 56
  30. 2024Investigating the Histogram Loss in Regressioncited by 11
  31. 2022Bilinear value networkscited by 9
  32. 2024Variance reduction of diffusion model's gradients with Taylor approximation-based control variatecited by 6
  33. 2025Of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agentscited by 2
  34. 2025ULTHO: Ultra-Lightweight yet Efficient Hyperparameter Optimization in Deep Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Actor-Critic without Actor
  2. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
  3. 2024Diffusion Actor-Critic with Entropy Regulator
  4. 2026ReBRAC-v2: The Return of the King
  5. 2026Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysis
  6. 2026SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer
  7. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  8. 2026Decoupling Policy Extraction for Offline Reinforcement Learning