Data to 13 Oct 2026

DIME:Diffusion-Based Maximum Entropy Reinforcement Learning

arXiv 2502.02316 ↗First posted 4 Feb 2025cs.LGcited by 53

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more

Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

Maximum entropy reinforcement learning (MaxEnt-RL) has become the standard approach to RL due to its beneficial exploration properties. Traditionally, policies are parameterized using Gaussian distributions, which significantly limits their representational capacity. Diffusion-based policies offer a more expressive alternative, yet integrating them into MaxEnt-RL poses challenges-primarily due to the intractability of computing their marginal entropy. To overcome this, we propose Diffusion-Based Maximum Entropy RL (DIME). DIME leverages recent advances in approximate inference with diffusion models to derive a lower bound on the maximum entropy objective. Additionally, we propose a policy iteration scheme that provably converges to the optimal diffusion policy. Our method enables the use of expressive diffusion-based policies while retaining the principled exploration benefits of MaxEnt-RL, significantly outperforming other diffusion-based methods on challenging high-dimensional control benchmarks. It is also competitive with state-of-the-art non-diffusion based RL methods while requiring fewer algorithmic design choices and smaller update-to-data ratios, reducing computational complexity.

Key facts. Cited by 53 papers in the map (data through 13 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 53 of its 76 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

53 of 76 references.

Closest ancestors · 12

Same direction, newest first

Other references · 41

2020Denoising Diffusion Probabilistic Modelscited by 20,952
2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,028
2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,023
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,023
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,889
2018Soft Actor-Critic Algorithms and Applicationscited by 1,185
2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,035
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 790
2017A Distributional Perspective on Reinforcement Learningcited by 745
2017Reinforcement Learning with Deep Energy-Based Policiescited by 660
2020dm_control: Software and Tasks for Continuous Controlcited by 369

29 more

2022The Primacy Bias in Deep Reinforcement Learningcited by 243
2021Path Integral Sampler: a stochastic control approach for samplingcited by 178
2017A unified view of entropy-regularized Markov decision processescited by 177
2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
2022An optimal control perspective on diffusion-based generative modelingcited by 146
2023Denoising Diffusion Samplerscited by 146
2024Iterated Denoising Energy Matching for Sampling from Boltzmann Densitiescited by 115
2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 106
2023Improved sampling via learned diffusionscited by 103
2019Theoretical guarantees for sampling and inference in generative models with latent diffusionscited by 97
2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 90
2015Hierarchical Variational Modelscited by 89
2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80
2016Auxiliary Deep Generative Modelscited by 72
2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 66
2023Diffusion Generative Flow Samplers: Improving learning signals through partial trajectory optimizationcited by 57
2021Bayesian Learning via Neural Schr"odinger-F"ollmer Flowscited by 56
2022Score-Based Diffusion meets Annealed Importance Samplingcited by 51
2024Sequential Controlled Langevin Diffusionscited by 47
2015The Variational Gaussian Processcited by 29
2022Langevin Diffusion Variational Inferencecited by 26
2016Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learningcited by 21
2023Transport meets Variational Inference: Controlled Monte Carlo Diffusionscited by 21
2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Criticcited by 18
2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 18
2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learningcited by 17
2024Variational Distillation of Diffusion Policies into Mixture of Expertscited by 14
2024Learned Reference-based Diffusion Sampling for multi-modal distributionscited by 9
2023On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economicscited by 6

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 17; 2026: 36.

53 papers.

Sort:

By area:

2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 27
2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
2025Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inferencecited by 23
2025One-Step Flow Policy Mirror Descentcited by 18
2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 16
2026Flow Policy Gradients for Robot Controlcited by 14
2025Enhanced DACER Algorithm with High Diffusion Efficiencycited by 13
2025Learning Boltzmann Generators via Constrained Mass Transportcited by 8
2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policiescited by 8
2025Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spacescited by 7
2025Relative Entropy Pathwise Policy Optimizationcited by 6

38 more

2026Verifier-Constrained Flow Expansion for Discovery Beyond the Datacited by 6
2026Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flowcited by 5
2026FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matchingcited by 5
2026SEAR: Sample Efficient Action Chunking Reinforcement Learningcited by 5
2025Exploratory Diffusion Model for Unsupervised Reinforcement Learningcited by 4
2025End-To-End Learning of Gaussian Mixture Priors for Diffusion Samplercited by 4
2025Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraintscited by 4
2026Reparameterization Flow Policy Optimizationcited by 4
2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
2025Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RLcited by 2
2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learningcited by 2
2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Controlcited by 2
2026Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policiescited by 2
2026What Does Flow Matching Bring To TD Learning?cited by 2
2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policiescited by 2
2026Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Samplingcited by 2
2026When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropycited by 2
2026FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidancecited by 2
2025Offline Reinforcement Learning with Generative Trajectory Policiescited by 1
2026Direct Soft-Policy Sampling via Langevin Dynamicscited by 1
2026GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learningcited by 1
2026ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priorscited by 1
2026Flow Matching Policy Optimization with Mirror Descent and Entropy Constraintscited by 1
2026Mean Flow Policy Optimizationcited by 1
2026Generative Actor-Critic with Soft Bridge Policiescited by 1
2026GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratioscited by 1
2026Active Flow Expansion for Out-of-Distribution Discovery: from Theory to Moleculescited by 1
2026Trust-Region Diffusion Policies for Massively Parallel On-Policy RLcited by 1
2025PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explorecited by 0
2026Latent Spherical Flow Policy for Reinforcement Learning with Combinatorial Actionscited by 0
2026Distributional Reinforcement Learning with Diffusion Bridge Criticscited by 0
2026Path-Space Mirror Descent for On-Policy Reinforcement Learning under the Generalized Schr"odinger Bridgecited by 0
2026Flow-based Policy With Distributional Reinforcement Learning in Trajectory Optimizationcited by 0
2026Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Datacited by 0
2026The Diffusion Encodercited by 0
2026Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descentcited by 0
2026Guided Discovery of New Behaviors using Diffusion Policiescited by 0
2026Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policycited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way