mlsanityData to 14 Oct 2026

A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning

arXiv 2410.23737 ↗First posted 31 Oct 2024cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline-to-online reinforcement learning (RL) leverages both pre-trained offline policies and online policies trained for downstream tasks, aiming to improve data efficiency and accelerate performance enhancement. An existing approach, Policy Expansion (PEX), utilizes a policy set composed of both policies without modifying the offline policy for exploration and learning. However, this approach fails to ensure sufficient learning of the online policy due to an excessive focus on exploration with both policies. Since the pre-trained offline policy can assist the online policy in exploiting a downstream task based on its prior experience, it should be executed effectively and tailored to the specific requirements of the downstream task. In contrast, the online policy, with its immature behavioral strategy, has the potential for exploration during the training phase. Therefore, our research focuses on harmonizing the advantages of the offline policy, termed exploitation, with those of the online policy, referred to as exploration, without modifying the offline policy. In this study, we propose an innovative offline-to-online RL method that employs a non-monolithic exploration approach. Our methodology demonstrates superior performance compared to PEX.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 30 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

26 of 30 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning13 Jun
  2. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  3. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  4. 2023Guiding Online Reinforcement Learning with Action-Free Offline Pretraining30 Jan
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  6. 2022MOORe: Model-based Offline-to-Online Reinforcement Learning25 Jan
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  9. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  10. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  11. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
4 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 10

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  4. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  5. 2022Online Decision Transformercited by 196
  6. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  7. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  8. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  9. 2022Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretrainingcited by 37
  10. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
  2. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  3. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  4. 2026Active Offline-to-Online Reinforcement Learning
  5. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learning
  6. 2024Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning
  7. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  8. 2022The Challenges of Exploration for Offline Reinforcement Learning