mlsanityData to 14 Oct 2026

Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning

arXiv 2310.05723 ↗First posted 9 Oct 2023cs.LGcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline pretraining with a static dataset followed by online fine-tuning (offline-to-online, or OtO) is a paradigm well matched to a real-world RL deployment process. In this scenario, we aim to find the best-performing policy within a limited budget of online interactions. Previous work in the OtO setting has focused on correcting for bias introduced by the policy-constraint mechanisms of offline RL algorithms. Such constraints keep the learned policy close to the behavior policy that collected the dataset, but we show this can unnecessarily limit policy performance if the behavior policy is far from optimal. Instead, we forgo constraints and frame OtO RL as an exploration problem that aims to maximize the benefit of online data-collection. We first study the major online RL exploration methods based on intrinsic rewards and UCB in the OtO setting, showing that intrinsic rewards add training instability through reward-function modification, and UCB methods are myopic and it is unclear which learned-component's ensemble to use for action selection. We then introduce an algorithm for planning to go out-of-distribution (PTGOOD) that avoids these issues. PTGOOD uses a non-myopic planning procedure that targets exploration in relatively high-reward regions of the state-action space unlikely to be visited by the behavior policy. By leveraging concepts from the Conditional Entropy Bottleneck, PTGOOD encourages data collected online to provide new information relevant to improving the final deployment policy without altering rewards. We show empirically in several continuous control tasks that PTGOOD significantly improves agent returns during online fine-tuning and avoids the suboptimal policy convergence that many of our baselines exhibit in several environments.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 47 of its 60 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

47 of 60 references.

Closest ancestors · 18

Same direction, newest first

  1. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning25 May
  2. 2023Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning17 May
  3. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions30 Mar
  4. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  6. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  7. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  8. 2022MOORe: Model-based Offline-to-Online Reinforcement Learning25 Jan
  9. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  10. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
6 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020MOPO: Model-based Offline Policy Optimization27 May
  3. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 29

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2018DeepMind Control Suitecited by 956
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  12. 2016Deep Exploration via Bootstrapped DQNcited by 527
17 more
  1. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  2. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  3. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  4. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  5. 2017Count-Based Exploration with Neural Density Modelscited by 257
  6. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  7. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  8. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  9. 2018Model-Based Active Explorationcited by 105
  10. 2020The Conditional Entropy Bottleneckcited by 75
  11. 2023Empirical Design in Reinforcement Learningcited by 55
  12. 2018Uncertainty in the Variational Information Bottleneckcited by 39
  13. 2021Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Explorationcited by 25
  14. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20
  15. 2019Explicit Explore-Exploit Algorithms in Continuous State Spacescited by 19
  16. 2022Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networkscited by 13
  17. 2023Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learningcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  2. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 8
  3. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  4. 2024SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  2. 2026Active Offline-to-Online Reinforcement Learning
  3. 2024A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning
  4. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithm
  5. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  6. 2025A Clean Slate for Offline Reinforcement Learning
  7. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learning
  8. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption