mlsanityData to 14 Oct 2026

Active Offline-to-Online Reinforcement Learning

arXiv 2607.11720 ↗First posted 13 Jul 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Background: Offline reinforcement learning (RL) enables effective policies to be trained from large, previously collected datasets and subsequently improved through limited online interaction. This offline-to-online RL (O2O-RL) paradigm is particularly promising in nonstationary domains where interaction is costly or potentially hazardous. Standard O2O-RL pipelines train multiple candidate policies offline, evaluate them using off-policy or online evaluation, and then deploy and fine-tune the policy with the highest estimated value. However, as in offline pretraining, fine-tuning performance is highly sensitive to the choice of algorithm and hyperparameters, making it risky to commit to a single policy. Objectives: We study active policy selection for fine-tuning under a limited interaction budget in O2O-RL settings. To our knowledge, this is the first work to address this problem. Methods: We formulate the problem by identifying a fundamental trade-off between allocating online interactions to policy evaluation, which helps identify high-performing policies, and allocating them to fine-tuning, which improves policy performance. We then propose an approach that balances this trade-off by actively selecting policies for fine-tuning based on upper-confidence bounds on their future performance. These bounds are derived from locally linear performance forecasts fitted to observations obtained through online evaluation. Results: Across a diverse range of experiments, the proposed approach consistently outperforms existing O2O-RL baselines. Conclusions: Actively selecting and fine-tuning policies uses limited online interaction budgets more effectively than either committing to a single policy or dividing the budget equally among all policies. Our framework also advances offline RL toward practical deployment in real-world systems where online interaction is costly or risky.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 31 references that are in the map. First posted on arXiv 13 Jul 2026.

Builds on

26 of 31 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning6 May
  2. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness29 Sep
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters8 Oct
  5. 2021Offline RL Without Off-Policy Evaluation16 Jun
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  8. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 15

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  7. 2018Behavioral Cloning from Observationcited by 492
  8. 2021Implicit Behavioral Cloningcited by 452
  9. 2014On the Complexity of Best Arm Identification in Multi-Armed Bandit Modelscited by 286
  10. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  11. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  12. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
3 more
  1. 2019Provably Efficient Q-Learning with Low Switching Costcited by 78
  2. 2021Active Offline Policy Selectioncited by 26
  3. 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  2. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  3. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
  4. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
  5. 2025A Clean Slate for Offline Reinforcement Learning
  6. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  7. 2025Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only
  8. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data