mlsanityData to 14 Oct 2026

An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutines

arXiv 2512.00383 ↗First posted 29 Nov 2025cs.LG cs.AI stat.MLcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We take the novel perspective of incorporating offline RL algorithms as subroutines of tabula rasa online RL. This is feasible because an online learning agent can repurpose its historical interactions as offline dataset. We formalize this idea into a framework that accommodates several variants of offline RL incorporation such as final policy recommendation and online fine-tuning. We further introduce convenient techniques to improve its effectiveness in enhancing online learning efficiency. Our extensive and systematic empirical analyses show that 1) the effectiveness of the proposed framework depends strongly on the nature of the task, 2) our proposed techniques greatly enhance its effectiveness, and 3) existing online fine-tuning methods are overall ineffective, calling for more research therein.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 25 references that are in the map. First posted on arXiv 29 Nov 2025.

Builds on

17 of 25 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting22 Jun
  2. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL7 Jun
  3. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  4. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  5. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  6. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  7. 2021A Dataset Perspective on Offline Reinforcement Learning8 Nov
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 7

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  3. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  5. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  6. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  7. 2021MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learningcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
  2. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
  3. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  4. 2025Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis
  5. 2026Active Offline-to-Online Reinforcement Learning
  6. 2021A Minimalist Approach to Offline Reinforcement Learning
  7. 2022Leveraging Offline Data in Online Reinforcement Learning
  8. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies