mlsanityData to 14 Oct 2026

Diffusion Actor-Critic with Entropy Regulator

arXiv 2405.15177 ↗First posted 24 May 2024cs.LG cs.AIcited by 94

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) has proven highly effective in addressing complex decision-making and control tasks. However, in most traditional RL algorithms, the policy is typically parameterized as a diagonal Gaussian distribution with learned mean and variance, which constrains their capability to acquire complex policies. In response to this problem, we propose an online RL algorithm termed diffusion actor-critic with entropy regulator (DACER). This algorithm conceptualizes the reverse process of the diffusion model as a novel policy function and leverages the capability of the diffusion model to fit multimodal distributions, thereby enhancing the representational capacity of the policy. Since the distribution of the diffusion policy lacks an analytical expression, its entropy cannot be determined analytically. To mitigate this, we propose a method to estimate the entropy of the diffusion policy utilizing Gaussian mixture model. Building on the estimated entropy, we can learn a parameter α that modulates the degree of exploration and exploitation. Parameter α will be employed to adaptively regulate the variance of the added noise, which is applied to the action output by the diffusion model. Experimental trials on MuJoCo benchmarks and a multimodal task demonstrate that the DACER algorithm achieves state-of-the-art (SOTA) performance in most MuJoCo control tasks while exhibiting a stronger representational capacity of the diffusion policy.

Key facts. Cited by 94 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 31 of its 44 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

31 of 44 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  2. 2023Boosting Continuous Control with Consistency Policy10 Oct
  3. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  4. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  6. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  7. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  12. 2015Trust Region Policy Optimizationcited by 2,580
12 more
  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2022Diffusion Models in Vision: A Surveycited by 1,015
  5. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  7. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 346
  8. 2022Diffusion Probabilistic Modeling for Video Generationcited by 238
  9. 2021Offline RL Without Off-Policy Evaluationcited by 151
  10. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learningcited by 91
  11. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 85
  12. 2021Integrated Decision and Control: Towards Interpretable and Computationally Efficient Driving Intelligencecited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 29; 2026: 61.

94 papers.

Sort:

By area:

  1. 2025Efficient Online Reinforcement Learning for Diffusion Policycited by 71
  2. 2025DeepAgent: A General Reasoning Agent with Scalable Toolsetscited by 71
  3. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 58
  4. 2025Flow-Based Policy for Online Reinforcement Learningcited by 37
  5. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 34
  6. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  7. 2024Schedule On the Fly: Diffusion Time Prediction for Faster and Better Image Generationcited by 30
  8. 2025One-Step Flow Policy Mirror Descentcited by 21
  9. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 20
  10. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 20
  11. 2025One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlowcited by 18
  12. 2024Sampling from Energy-based Policies using Diffusioncited by 17
38 more
  1. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learningcited by 17
  2. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  3. 2025Enhanced DACER Algorithm with High Diffusion Efficiencycited by 13
  4. 2025From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learningcited by 12
  5. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policiescited by 12
  6. 2025D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learningcited by 10
  7. 2026Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generationcited by 9
  8. 2026STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokenscited by 9
  9. 2025Habitizing Diffusion Planning for Efficient and Effective Decision Makingcited by 7
  10. 2025Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spacescited by 7
  11. 2025CHD: Coupled Hierarchical Diffusion for Long-Horizon Taskscited by 6
  12. 2026FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matchingcited by 6
  13. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 6
  14. 2026Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flowcited by 5
  15. 2026PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learningcited by 5
  16. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?cited by 5
  17. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
  18. 2025Adaptive Diffusion Policy Optimization for Robotic Manipulationcited by 4
  19. 2025Distributional Soft Actor-Critic with Diffusion Policycited by 4
  20. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 4
  21. 2026Reparameterization Flow Policy Optimizationcited by 4
  22. 2024AERO: Entropy-Guided Framework for Private LLM Inferencecited by 3
  23. 2025Off-policy Reinforcement Learning with Model-based Exploration Augmentationcited by 3
  24. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3
  25. 2026TourPlanner: A Competitive Consensus Framework with Constraint-Gated Reinforcement Learning for Travel Planningcited by 3
  26. 2026Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policiescited by 3
  27. 2026GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policiescited by 3
  28. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policiescited by 3
  29. 2026Mean Flow Policy Optimizationcited by 3
  30. 2026FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidancecited by 3
  31. 2026GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratioscited by 3
  32. 2025Confidence-Regulated Generative Diffusion Models for Reliable AI Agent Migration in Vehicular Metaversescited by 2
  33. 2025Consistency Trajectory Planning: High-Quality and Efficient Trajectory Optimization for Offline Model-Based Reinforcement Learningcited by 2
  34. 2025Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RLcited by 2
  35. 2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Controlcited by 2
  36. 2026Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matchingcited by 2
  37. 2026Diffusion Policy through Conditional Proximal Policy Optimizationcited by 2
  38. 2026Flow Matching Policy Optimization with Mirror Descent and Entropy Constraintscited by 2

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
  2. 2025Dichotomous Diffusion Policy Optimization
  3. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient
  4. 2025D2 Actor Critic: Diffusion Actor Meets Distributional Critic
  5. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning
  6. 2026Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysis
  7. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization
  8. 2024Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models