mlsanityData to 14 Oct 2026

A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning

arXiv 2306.07541 ↗First posted 13 Jun 2023cs.LG cs.AIcited by 8

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) provides a promising solution to learning an agent fully relying on a data-driven paradigm. However, constrained by the limited quality of the offline dataset, its performance is often sub-optimal. Therefore, it is desired to further finetune the agent via extra online interactions before deployment. Unfortunately, offline-to-online RL can be challenging due to two main challenges: constrained exploratory behavior and state-action distribution shift. In view of this, we propose a Simple Unified uNcertainty-Guided (SUNG) framework, which naturally unifies the solution to both challenges with the tool of uncertainty. Specifically, SUNG quantifies uncertainty via a VAE-based state-action visitation density estimator. To facilitate efficient exploration, SUNG presents a practical optimistic exploration strategy to select informative actions with both high value and high uncertainty. Moreover, SUNG develops an adaptive exploitation method by applying conservative offline RL objectives to high-uncertainty samples and standard online RL objectives to low-uncertainty samples to smoothly bridge offline and online stages. SUNG achieves state-of-the-art online finetuning performance when combined with different offline RL methods, across various environments and datasets in D4RL benchmark. Codes are made publicly available in https://github.com/guosyjlu/SUNG.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 57 of its 88 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

57 of 88 references.

Closest ancestors · 31

Same direction, newest first

  1. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles12 Jun
  2. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning25 May
  3. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions30 Mar
  4. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  5. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  7. 2023Behavior Proximal Policy Optimization22 Feb
  8. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  9. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning25 Oct
  10. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief13 Oct
  11. 2022Offline Reinforcement Learning at Multiple Frequencies26 Jul
  12. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
19 more
  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  2. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  4. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  5. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  6. 2022MOORe: Model-based Offline-to-Online Reinforcement Learning25 Jan
  7. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  10. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  13. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  14. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  15. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  16. 2020MOPO: Model-based Offline Policy Optimization27 May
  17. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  18. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  19. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 26

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2019An Introduction to Variational Autoencoderscited by 1,000
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  10. 2016Deep Exploration via Bootstrapped DQNcited by 527
  11. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  12. 2015Importance Weighted Autoencoderscited by 456
14 more
  1. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  2. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  3. 2022Online Decision Transformercited by 196
  4. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  5. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  6. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  7. 2018Uncertainty in Neural Networks: Approximately Bayesian Ensemblingcited by 94
  8. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  9. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  10. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  11. 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 40
  12. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  13. 2023User Retention-oriented Recommendation with Decision Transformercited by 14
  14. 2023Clinical Decision Transformer: Intended Treatment Recommendation through Goal Promptingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 1; 2026: 1.

8 papers.

  1. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
  2. 2024Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modelingcited by 18
  3. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  4. 2025Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generationcited by 9
  5. 2023Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Explorationcited by 0
  6. 2024SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidancecited by 0
  7. 2024A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learningcited by 0
  8. 2026SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning
  2. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  3. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  4. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  5. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  6. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  7. 2023Selective Uncertainty Propagation in Offline RL
  8. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning