mlsanityData to 14 Oct 2026

Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning

arXiv 2304.12824 ↗First posted 25 Apr 2023cs.LGcited by 149

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Guided sampling is a vital approach for applying diffusion models in real-world tasks that embeds human-defined guidance during the sampling procedure. This paper considers a general setting where the guidance is defined by an (unnormalized) energy function. The main challenge for this setting is that the intermediate guidance during the diffusion sampling procedure, which is jointly defined by the sampling distribution and the energy function, is unknown and is hard to estimate. To address this challenge, we propose an exact formulation of the intermediate guidance as well as a novel training objective named contrastive energy prediction (CEP) to learn the exact guidance. Our method is guaranteed to converge to the exact guidance under unlimited model capacity and data samples, while previous methods can not. We demonstrate the effectiveness of our method by applying it to offline reinforcement learning (RL). Extensive experiments on D4RL benchmarks demonstrate that our method outperforms existing state-of-the-art algorithms. We also provide some examples of applying CEP for image synthesis to demonstrate the scalability of CEP on high-dimensional data.

Key facts. Cited by 149 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 67 of its 85 references that are in the map. First posted on arXiv 25 Apr 2023.

Builds on

67 of 85 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2022Generative Adversarial Networkscited by 5,627
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2022Video Diffusion Modelscited by 2,305
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  12. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,416
  13. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  14. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  15. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  16. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  17. 2021Variational Diffusion Modelscited by 1,153
  18. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  19. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  20. 2022Denoising Diffusion Restoration Modelscited by 963
  21. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  22. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  23. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  24. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  25. 2022On Distillation of Guided Diffusion Modelscited by 725
  26. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 677
  27. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  28. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  29. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  30. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 532
  31. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 516
  32. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  33. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 334
  34. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusioncited by 309
  35. 2020Critic Regularized Regressioncited by 275
  36. 2022Diffusion Probabilistic Modeling for Video Generationcited by 238
  37. 2022Diffusion models as plug-and-play priorscited by 231
  38. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 31; 2025: 52; 2026: 57.

149 papers.

Sort:

By area:

  1. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 230
  2. 2025Flow Q-Learningcited by 184
  3. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  4. 2025Diffusion-Based Planning for Autonomous Driving with Flexible Guidancecited by 141
  5. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 138
  6. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 110
  7. 2024TFG: Unified Training-Free Guidance for Diffusion Modelscited by 107
  8. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 100
  9. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  10. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  11. 2024Simple Hierarchical Planning with Diffusioncited by 73
  12. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 73
38 more
  1. 2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 72
  2. 2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 71
  3. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 71
  4. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
  5. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 59
  6. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 58
  7. 2023Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimizationcited by 56
  8. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 55
  9. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 49
  10. 2025On the Guidance of Flow Matchingcited by 45
  11. 2024Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Followingcited by 42
  12. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  13. 2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 39
  14. 2023Boosting Continuous Control with Consistency Policycited by 38
  15. 2025What Makes a Good Diffusion Planner for Decision Making?cited by 38
  16. 2025Flow-Based Policy for Online Reinforcement Learningcited by 37
  17. 2026Q-learning with Adjoint Matchingcited by 37
  18. 2024Diffusion Models as Constrained Samplers for Optimization with Unknown Constraintscited by 35
  19. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 34
  20. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 34
  21. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 34
  22. 2024Understanding and Improving Training-free Loss-based Diffusion Guidancecited by 32
  23. 2025Contrastive Flow Matchingcited by 29
  24. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learningcited by 28
  25. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  26. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 20
  27. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 20
  28. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimizationcited by 19
  29. 2024Sampling from Energy-based Policies using Diffusioncited by 17
  30. 2026Flow Policy Gradients for Robot Controlcited by 16
  31. 2023Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RLcited by 15
  32. 2025FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learningcited by 15
  33. 2024Toward Guidance-Free AR Visual Generation via Condition Contrastive Alignmentcited by 14
  34. 2025Skill Expansion and Composition in Parameter Spacecited by 14
  35. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learningcited by 14
  36. 2023Efficient Planning with Latent Diffusioncited by 12
  37. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Controlcited by 12
  38. 2025Discrete Diffusion Trajectory Alignment via Stepwise Decompositioncited by 12

50 most cited of 149 All 149 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning
  2. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?
  3. 2025Energy-Based Transfer for Reinforcement Learning
  4. 2023Generalized Contrastive Divergence: Joint Training of Energy-Based Model and Diffusion Model through Inverse Reinforcement Learning
  5. 2026Manifold-Constrained Energy-Based Transition Models for Offline Reinforcement Learning
  6. 2024Diffusion Spectral Representation for Reinforcement Learning
  7. 2026Improving Diffusion Planners by Self-Supervised Action Gating with Energies
  8. 2024Off-dynamics Conditional Diffusion Planners