mlsanityData to 14 Oct 2026

Sampling from Energy-based Policies using Diffusion

arXiv 2410.01312 ↗First posted 2 Oct 2024cs.LGcited by 17

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

Energy-based policies offer a flexible framework for modeling complex, multimodal behaviors in reinforcement learning (RL). In maximum entropy RL, the optimal policy is a Boltzmann distribution derived from the soft Q-function, but direct sampling from this distribution in continuous action spaces is computationally intractable. As a result, existing methods typically use simpler parametric distributions, like Gaussians, for policy representation -- limiting their ability to capture the full complexity of multimodal action distributions. In this paper, we introduce a diffusion-based approach for sampling from energy-based policies, where the negative Q-function defines the energy function. Based on this approach, we propose an actor-critic method called Diffusion Q-Sampling (DQS) that enables more expressive policy representations, allowing stable learning in diverse environments. We show that our approach enhances sample efficiency in continuous control tasks and captures multimodal behaviors, addressing key limitations of existing methods. Code is available at https://github.com/vineetjain96/Diffusion_Q_Sampling.git

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 41 of its 50 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

41 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Diffusion Actor-Critic with Entropy Regulator24 May

Other references · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  10. 2016Density estimation using Real NVPcited by 2,042
  11. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
28 more
  1. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  2. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  6. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  7. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  8. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 392
  9. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  10. 2021GFlowNet Foundationscited by 290
  11. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 251
  12. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 236
  13. 2021Gotta Go Fast When Generating Data with Score-Based Modelscited by 219
  14. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 157
  15. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 149
  16. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 138
  17. 2022Learning GFlowNets from partial episodes for improved convergence and stabilitycited by 132
  18. 2024Iterated Denoising Energy Matching for Sampling from Boltzmann Densitiescited by 116
  19. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 112
  20. 2018The Limits and Potentials of Deep Learning for Roboticscited by 104
  21. 2023A theory of continuous generative flow networkscited by 101
  22. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 100
  23. 2023Towards Understanding and Improving GFlowNet Trainingcited by 75
  24. 2023Generative Flow Networks as Entropy-Regularized RLcited by 64
  25. 2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 44
  26. 2023Thompson sampling for improved exploration in GFlowNetscited by 32
  27. 2023Learning to Reach Goals via Diffusioncited by 10
  28. 2021Exponential Family Model-Based Reinforcement Learning via Score Matchingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 12.

17 papers.

Sort:

By area:

  1. 2025Efficient Online Reinforcement Learning for Diffusion Policycited by 71
  2. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 34
  3. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policiescited by 12
  4. 2026FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matchingcited by 6
  5. 2026Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flowcited by 5
  6. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
  7. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3
  8. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policiescited by 3
  9. 2025Consistency Trajectory Planning: High-Quality and Efficient Trajectory Optimization for Offline Model-Based Reinforcement Learningcited by 2
  10. 2025Actor-Critic without Actorcited by 1
  11. 2026Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learningcited by 1
  12. 2026Dual-Flow Reinforcement Learning with State-Aware Explorationcited by 1
5 more
  1. 2026CHDP: Cooperative Hybrid Diffusion Policies for Reinforcement Learning in Parameterized Action Spacecited by 0
  2. 2026Flow Perturbation++: Multi-Step Unbiased Jacobian Estimation for High-Dimensional Boltzmann Samplingcited by 0
  3. 2026Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitionerscited by 0
  4. 2026Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learningcited by 0
  5. 2026Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?
  2. 2026Diffusion Policy through Conditional Proximal Policy Optimization
  3. 2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
  4. 2024Diffusion Spectral Representation for Reinforcement Learning
  5. 2026Direct Soft-Policy Sampling via Langevin Dynamics
  6. 2023Score Regularized Policy Optimization through Diffusion Behavior
  7. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient
  8. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization