mlsanityData to 7 Oct 2026

MOORe: Model-based Offline-to-Online Reinforcement Learning

arXiv 2201.10070 ↗First posted 25 Jan 2022cs.LGcited by 9

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

With the success of offline reinforcement learning (RL), offline trained RL policies have the potential to be further improved when deployed online. A smooth transfer of the policy matters in safe real-world deployment. Besides, fast adaptation of the policy plays a vital role in practical online performance improvement. To tackle these challenges, we propose a simple yet efficient algorithm, Model-based Offline-to-Online Reinforcement learning (MOORe), which employs a prioritized sampling scheme that can dynamically adjust the offline and online data for smooth and efficient online adaptation of the policy. We provide a theoretical foundation for our algorithms design. Experiment results on the D4RL benchmark show that our algorithm smoothly transfers from offline to online stages while enabling sample-efficient online adaption, and also significantly outperforms existing methods.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 8 of its 22 references that are in the map. First posted on arXiv 25 Jan 2022.

Builds on

8 of 22 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 5

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2015Prioritized Experience Replaycited by 1,070
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 4.

9 papers.

  1. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  2. 2023Large Sequence Models for Sequential Decision-Making: A Surveycited by 19
  3. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  4. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  5. 2024DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delayscited by 6
  6. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learningcited by 4
  7. 2022Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemblecited by 1
  8. 2024SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidancecited by 0
  9. 2024A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Clean Slate for Offline Reinforcement Learning
  2. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learning
  3. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning
  4. 2020MOPO: Model-based Offline Policy Optimization
  5. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  6. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  7. 2021A Minimalist Approach to Offline Reinforcement Learning
  8. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model