mlsanityData to 7 Oct 2026

Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning

arXiv 2402.04080 ↗First posted 6 Feb 2024cs.LG cs.SY eess.SYcited by 29

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

This paper presents advanced techniques of training diffusion policies for offline reinforcement learning (RL). At the core is a mean-reverting stochastic differential equation (SDE) that transfers a complex action distribution into a standard Gaussian and then samples actions conditioned on the environment state with a corresponding reverse-time SDE, like a typical diffusion policy. We show that such an SDE has a solution that we can use to calculate the log probability of the policy, yielding an entropy regularizer that improves the exploration of offline datasets. To mitigate the impact of inaccurate value functions from out-of-distribution data points, we further propose to learn the lower confidence bound of Q-ensembles for more robust policy improvement. By combining the entropy-regularized diffusion policy with Q-ensembles in offline RL, our method achieves state-of-the-art performance on most tasks in D4RL benchmarks. Code is available at https://github.com/ruoqizzz/Entropy-Regularized-Diffusion-Policy-with-QEnsemble.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 40 of its 52 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

40 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Diffusion Models for Reinforcement Learning: A Survey2 Nov
  2. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  3. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  4. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 36

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  12. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
24 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  7. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  10. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  11. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  12. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  13. 2016Deep Exploration via Bootstrapped DQNcited by 518
  14. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  15. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  16. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Modelscited by 195
  17. 2023Image Restoration with Mean-Reverting Stochastic Differential Equationscited by 182
  18. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  19. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  20. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 103
  21. 2022Speech Enhancement with Score-Based Generative Models in the Complex STFT Domaincited by 102
  22. 2023Refusion: Enabling Large-Size Realistic Image Restoration with Latent-Space Diffusion Modelscited by 86
  23. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  24. 2019Learning to Combat Compounding-Error in Model-Based Reinforcement Learningcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 15.

29 papers.

Sort:

By area:

  1. 2025Flow Q-Learningcited by 162
  2. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  3. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  4. 2026Q-learning with Adjoint Matchingcited by 33
  5. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  6. 2024Photo-Realistic Image Restoration in the Wild with Controlled Vision-Language Modelscited by 19
  7. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 18
  8. 2026Flow Policy Gradients for Robot Controlcited by 14
  9. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learningcited by 13
  10. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learningcited by 8
  11. 2025Unleashing Flow Policies with Distributional Criticscited by 7
  12. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
17 more
  1. 2024The Exploration-Exploitation Dilemma Revisited: An Entropy Perspectivecited by 5
  2. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 5
  3. 2025Real-Time Diffusion Policies for Games: Enhancing Consistency Policies with Q-Ensemblescited by 3
  4. 2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policiescited by 2
  5. 2026Reversal Q-Learningcited by 2
  6. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learningcited by 1
  7. 2025Improving Sample Efficiency in Multi-Agent Reinforcement Learning for Simulated Football Games via Explorationcited by 1
  8. 2026Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Drivingcited by 1
  9. 2026Improving Diffusion Planners by Self-Supervised Action Gating with Energiescited by 1
  10. 2026Epistemic Robust Offline Reinforcement Learningcited by 1
  11. 2026Entropy-Regularized Adjoint Matching for Offline Reinforcement Learningcited by 1
  12. 2026Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learningcited by 1
  13. 2025Ensemble RL through Classifier Models: Enhancing Risk-Return Trade-offs in Trading Strategiescited by 0
  14. 2026Aligning Flow Map Policies with Optimal Q-Guidancecited by 0
  15. 2026Reinforcement Learning for Flow-Matching Policies with Density Transportcited by 0
  16. 2026VINE: Taming Generative Control Policies for Reinforcement Learningcited by 0
  17. 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  2. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning
  3. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
  4. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  5. 2026Diffusion Policy through Conditional Proximal Policy Optimization
  6. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies
  7. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching
  8. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning