mlsanityData to 14 Oct 2026

Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning

arXiv 2407.12448 ↗First posted 17 Jul 2024cs.LGcited by 17

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Combining offline and online reinforcement learning (RL) techniques is indeed crucial for achieving efficient and safe learning where data acquisition is expensive. Existing methods replay offline data directly in the online phase, resulting in a significant challenge of data distribution shift and subsequently causing inefficiency in online fine-tuning. To address this issue, we introduce an innovative approach, Energy-guided DIffusion Sampling (EDIS), which utilizes a diffusion model to extract prior knowledge from the offline dataset and employs energy functions to distill this knowledge for enhanced data generation in the online phase. The theoretical analysis demonstrates that EDIS exhibits reduced suboptimality compared to solely utilizing online data or directly reusing offline data. EDIS is a plug-in approach and can be combined with existing methods in offline-to-online RL setting. By implementing EDIS to off-the-shelf methods Cal-QL and IQL, we observe a notable 20% average improvement in empirical performance on MuJoCo, AntMaze, and Adroit environments. Code is available at urlhttps://github.com/liuxhym/EDIS.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 51 of its 60 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

51 of 60 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Policy-Guided Diffusion9 Apr
  2. 2023World Models via Policy-Guided Trajectory Diffusion13 Dec
  3. 2023Value function estimation using conditional diffusion models for control9 Jun
  4. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 46

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  11. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  12. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
34 more
  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  2. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  4. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  5. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  6. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  7. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  8. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  9. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 251
  10. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  11. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  12. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  13. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  14. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  15. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
  16. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
  17. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  18. 2023Synthetic Experience Replaycited by 101
  19. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 84
  20. 2018Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learningcited by 71
  21. 2020OR-Gym: A Reinforcement Learning Library for Operations Research Problemscited by 51
  22. 2023Learning Cut Selection for Mixed-Integer Linear Programming via Hierarchical Sequence Modelcited by 45
  23. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  24. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  25. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 29
  26. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  27. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  28. 2021Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Criticcited by 23
  29. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learningcited by 14
  30. 2022Adversarial Counterfactual Environment Model Learningcited by 12
  31. 2024Learning to Cut via Hierarchical Sequence/Set Model for Efficient Mixed-Integer Programmingcited by 12
  32. 2024Learning to Stop Cut Generation for Efficient Mixed-Integer Linear Programmingcited by 8
  33. 2022Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learningcited by 3
  34. 2024Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 6.

17 papers.

Sort:

By area:

  1. 2025Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generationcited by 9
  2. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6
  3. 2025Exploratory Diffusion Model for Unsupervised Reinforcement Learningcited by 5
  4. 2025The Three Regimes of Offline-to-Online Reinforcement Learningcited by 4
  5. 2025Wavelet Fourier Diffuser: Frequency-Aware Diffusion Model for Reinforcement Learningcited by 3
  6. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruptioncited by 3
  7. 2025Adaptive Replay Buffer for Offline-to-Online Reinforcement Learningcited by 3
  8. 2025Generative Actor Criticcited by 2
  9. 2025Offline Reinforcement Learning with Generative Trajectory Policiescited by 1
  10. 2026Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learningcited by 1
  11. 2024SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidancecited by 0
  12. 2025Trajectory First: A Curriculum for Discovering Diverse Policiescited by 0
5 more
  1. 2025From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learningcited by 0
  2. 2026ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Networkcited by 0
  3. 2026Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learningcited by 0
  4. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learningcited by 0
  5. 2026Revisiting TD Target Aggregation under Uncertainty in Q-Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning
  2. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  3. 2023Efficient Diffusion Policies for Offline Reinforcement Learning
  4. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  5. 2025A Clean Slate for Offline Reinforcement Learning
  6. 2022Leveraging Offline Data in Online Reinforcement Learning
  7. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
  8. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies