mlsanityData to 7 Oct 2026

Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning

arXiv 2605.30776 ↗First posted 29 May 2026cs.LGcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Offline-to-Online Reinforcement Learning (O2O-RL) leverages an offline, pre-trained policy to minimize costly online interactions. Although data-efficient, O2O-RL is susceptible to shifts between offline and online distributions. Existing work aims to mitigate the harm of this shift by finetuning the policy on trajectory data sampled from a diffusion model. Inspired by this line of work, we propose DUAL: an efficient Diffusion Uncertainty-Aware framework for offline-to-online reinforcement Learning. DUAL utilizes the prior knowledge of the diffusion model to distill a fast-sampling diffusion actor policy and transition model in the offline phase. DUAL also employs a Laplace approximation and distance transition-state-shift detection, thereby using uncertainty quantification to improve exploration versus exploitation in the online phase. We formally show that our actor loss with the Laplace approximation provides a proxy for a principled estimate of epistemic uncertainty. Empirically, DUAL improves the online expected return over O2O-RL baselines across multiple settings and environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 49 of its 60 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

49 of 60 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Adversarial Diffusion for Robust Reinforcement Learning28 Sep
  2. 2025What Makes a Good Diffusion Planner for Decision Making?1 Mar
  3. 2025Exploratory Diffusion Model for Unsupervised Reinforcement Learning11 Feb
  4. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning17 Jul
  5. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning30 May
  6. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  7. 2024Diffusion Actor-Critic with Entropy Regulator24 May
  8. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  9. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL31 May
  10. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  11. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  12. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 37

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  12. 2017A Distributional Perspective on Reinforcement Learningcited by 743
25 more
  1. 2016Deep Exploration via Bootstrapped DQNcited by 518
  2. 2018Maximum a Posteriori Policy Optimisationcited by 318
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  5. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  6. 2021Epistemic Neural Networkscited by 120
  7. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  8. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  9. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  10. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
  11. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  12. 2023Movement Primitive Diffusion: Learning Gentle Robotic Manipulation of Deformable Objectscited by 40
  13. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  14. 2023BayesDiff: Estimating Pixel-wise Uncertainty in Diffusion via Bayesian Inferencecited by 30
  15. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26
  16. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  17. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  18. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
  19. 2025Generative Uncertainty in Diffusion Modelscited by 20
  20. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learningcited by 15
  21. 2025Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generationcited by 9
  22. 2024Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shiftscited by 7
  23. 2024Variance-Aware Linear UCB with Deep Representation for Neural Contextual Banditscited by 3
  24. 2025Q-Learning with Shift-Aware Upper Confidence Bound in Non-Stationary Reinforcement Learningcited by 1
  25. 2026Beyond Penalization: Diffusion-based Out-of-Distribution Detection and Selective Regularization in Offline Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning
  2. 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning
  3. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  4. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  5. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  6. 2026Information-Directed Offline-to-Online Reinforcement Learning
  7. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  8. 2023Selective Uncertainty Propagation in Offline RL