mlsanityData to 14 Oct 2026

Maximum Entropy Reinforcement Learning with Diffusion Policy

arXiv 2502.11612 ↗First posted 17 Feb 2025cs.LG cs.AIcited by 34

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

The Soft Actor-Critic (SAC) algorithm with a Gaussian policy has become a mainstream implementation for realizing the Maximum Entropy Reinforcement Learning (MaxEnt RL) objective, which incorporates entropy maximization to encourage exploration and enhance policy robustness. While the Gaussian policy performs well on simpler tasks, its exploration capacity and potential performance in complex multi-goal RL environments are limited by its inherent unimodality. In this paper, we employ the diffusion model, a powerful generative model capable of capturing complex multimodal distributions, as the policy representation to fulfill the MaxEnt RL objective, developing a method named MaxEnt RL with Diffusion Policy (MaxEntDP). Our method enables efficient exploration and brings the policy closer to the optimal MaxEnt policy. Experimental results on Mujoco benchmarks show that MaxEntDP outperforms the Gaussian policy and other generative models within the MaxEnt RL framework, and performs comparably to other state-of-the-art diffusion-based online RL algorithms. Our code is available at https://github.com/diffusionyes/MaxEntDP.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 43 of its 54 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

43 of 54 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning29 Jul
  2. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control12 Jul
  3. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient2 Jun
  4. 2024Model-Based Diffusion for Trajectory Optimization28 May
  5. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  6. 2024Diffusion Actor-Critic with Entropy Regulator24 May
  7. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  8. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  9. 2023Boosting Continuous Control with Consistency Policy10 Oct
  10. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  11. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  12. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
3 more
  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 28

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Denoising Diffusion Implicit Modelscited by 9,781
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2022Generative Adversarial Networkscited by 5,627
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  9. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  11. 2022Video Diffusion Modelscited by 2,305
  12. 2015Variational Inference with Normalizing Flowscited by 2,170
16 more
  1. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  2. 2023Consistency Modelscited by 1,822
  3. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  5. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  6. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  7. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  8. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statisticscited by 161
  9. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Modelscited by 151
  10. 2024Iterated Denoising Energy Matching for Sampling from Boltzmann Densitiescited by 116
  11. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  12. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  13. 2024Improved off-policy training of diffusion samplerscited by 48
  14. 2023Information-Theoretic Diffusioncited by 44
  15. 2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flowcited by 26
  16. 2024Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Trainingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 28.

34 papers.

Sort:

By area:

  1. 2026Maximum Likelihood Reinforcement Learningcited by 36
  2. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 31
  3. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 20
  4. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  5. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policiescited by 12
  6. 2026Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flowcited by 5
  7. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?cited by 5
  8. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
  9. 2026Reparameterization Flow Policy Optimizationcited by 4
  10. 2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learningcited by 3
  11. 2026Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policiescited by 3
  12. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policiescited by 3
22 more
  1. 2026Mean Flow Policy Optimizationcited by 3
  2. 2026FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidancecited by 3
  3. 2026GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratioscited by 3
  4. 2026Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matchingcited by 2
  5. 2026Diffusion Policy through Conditional Proximal Policy Optimizationcited by 2
  6. 2026Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Samplingcited by 2
  7. 2026Fisher Decorator: Refining Flow Policy via a Local Transport Mapcited by 2
  8. 2025Accelerating Inference of Masked Image Generators via Reinforcement Learningcited by 1
  9. 2026Contractive Diffusion Policies: Robust Action Diffusion via Contractive Score-Based Sampling with Differential Equationscited by 1
  10. 2026Direct Soft-Policy Sampling via Langevin Dynamicscited by 1
  11. 2026GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learningcited by 1
  12. 2026NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learningcited by 1
  13. 2026Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learningcited by 1
  14. 2025D2 Actor Critic: Diffusion Actor Meets Distributional Criticcited by 0
  15. 2026Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descentcited by 0
  16. 2026Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimizationcited by 0
  17. 2026Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policycited by 0
  18. 2026Safe Score Matching: Diffusion Policies with Hamilton-Jacobi Reachability for Online Safe Reinforcement Learningcited by 0
  19. 2026Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learningcited by 0
  20. 2026Reinforcement Learning via Fixed-point Flow Matchingcited by 0
  21. 2026CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidancecited by 0
  22. 2026Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learning
  2. 2026Generative Actor-Critic with Soft Bridge Policies
  3. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  4. 2024Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimation
  5. 2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic
  6. 2025Max-Entropy Reinforcement Learning with Flow Matching and A Case Study on LQR
  7. 2025When Maximum Entropy Misleads Policy Optimization
  8. 2024Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models