mlsanityData to 7 Oct 2026

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

arXiv 2303.07693 ↗First posted 14 Mar 2023cs.LG cs.AIcited by 40

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Conventional reinforcement learning (RL) needs an environment to collect fresh data, which is impractical when online interactions are costly. Offline RL provides an alternative solution by directly learning from the previously collected dataset. However, it will yield unsatisfactory performance if the quality of the offline datasets is poor. In this paper, we consider an offline-to-online setting where the agent is first learned from the offline dataset and then trained online, and propose a framework called Adaptive Policy Learning for effectively taking advantage of offline and online data. Specifically, we explicitly consider the difference between the online and offline data and apply an adaptive update scheme accordingly, that is, a pessimistic update strategy for the offline dataset and an optimistic/greedy update scheme for the online dataset. Such a simple and effective method provides a way to mix the offline and online RL and achieve the best of both worlds. We further provide two detailed algorithms for implementing the framework through embedding value or policy-based RL algorithms into it. Finally, we conduct extensive experiments on popular continuous control tasks, and results show that our algorithm can learn the expert policy with high sample efficiency even when the quality of offline dataset is poor, e.g., random dataset.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 35 references that are in the map. First posted on arXiv 14 Mar 2023.

Builds on

25 of 35 references.

Closest ancestors · 11

Same direction, newest first

  1. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2020POPO: Pessimistic Offline Policy Optimization26 Dec
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOPO: Model-based Offline Policy Optimization27 May
  8. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
2 more
  1. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  2. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 7; 2025: 16; 2026: 10.

40 papers.

Sort:

By area:

  1. 2025Reinforcement Learning with Action Chunkingcited by 97
  2. 2026Q-learning with Adjoint Matchingcited by 33
  3. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  4. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  5. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  6. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  7. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  8. 2025Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generationcited by 9
  9. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  10. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  11. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  12. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learningcited by 7
28 more
  1. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Datacited by 5
  2. 2025Balancing optimism and pessimism in offline-to-online learningcited by 4
  3. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learningcited by 4
  4. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptationcited by 4
  5. 2026Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Actioncited by 4
  6. 2023Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approachcited by 3
  7. 2025Adaptive Replay Buffer for Offline-to-Online Reinforcement Learningcited by 3
  8. 2025Generative Machine Learning in Adaptive Control of Dynamic Manufacturing Processes: A Reviewcited by 2
  9. 2025Hybrid Combinatorial Multi-armed Bandits with Probabilistically Triggered Armscited by 2
  10. 2026RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Rankingcited by 2
  11. 2026FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillationcited by 2
  12. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithmscited by 1
  13. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbationscited by 1
  14. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspectivecited by 1
  15. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RLcited by 1
  16. 2023Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Explorationcited by 0
  17. 2024SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidancecited by 0
  18. 2024A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learningcited by 0
  19. 2025SLA-MORL: SLA-Aware Multi-Objective Reinforcement Learning for HPC Resource Optimizationcited by 0
  20. 2025Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Controlcited by 0
  21. 2025From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learningcited by 0
  22. 2025An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutinescited by 0
  23. 2026SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfercited by 0
  24. 2026ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimizationcited by 0
  25. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithmcited by 0
  26. 2026Information-Directed Offline-to-Online Reinforcement Learningcited by 0
  27. 2026Expert Behavior Prior Reinforcement Learningcited by 0
  28. 2026Augmenting Human Performance with an XR Agent Learning from Online Behavior and BCI Evidencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  2. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  3. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning
  4. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  5. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  6. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  7. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model
  8. 2026Active Offline-to-Online Reinforcement Learning