COOPO: Cyclic Offline-Online Policy Optimization Algorithm
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Offline reinforcement learning struggles with distributional shift and constrained performance due to static dataset limitations, while online RL demands prohibitive environment interactions. The recent advent of hybrid offline-to-online methods bridges these domains but suffers from distribution drift during transitions and catastrophic forgetting of offline knowledge. We introduce COOPO (Cyclic Offline-Online Policy Optimization), a generalized framework that repeatedly cycles between constrained offline training and online fine-tuning. Each cycle first anchors the policy to the dataset via KL-regularized advantage-weighted offline updates to minimize distributional shift and then fine-tunes it online using any policy optimization for stable exploration. Crucially, periodically returning to offline training eliminates forgetting and drift while maximizing dataset reuse. The cyclic behavior also helps reduce the online environment interactions. Theoretically, COOPO achieves better online sample efficiency, surpassing pure online RL, with guaranteed monotonic improvement under standard coverage assumptions. Extensive D4RL benchmarks demonstrate COOPO reduces online interactions versus state-of-the-art hybrids while improving final returns, maintaining robustness across diverse offline algorithms and online optimizers. This looped synergy sets new efficiency and performance standards for adaptive RL.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 55 references that are in the map. First posted on arXiv 18 May 2026.
Builds on
40 of 55 references.
Closest ancestors · 22
Same direction, newest first
- 2025Reinforcement Learning with Action Chunking10 Jul
- 2025Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only22 May
- 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL25 Dec
- 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data10 Dec
- 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization6 Nov
- 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles12 Jun
- 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions30 Mar
- 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
- 2023Behavior Proximal Policy Optimization22 Feb
- 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
- 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
10 more
- 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
- 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
- 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
- 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
- 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020MOPO: Model-based Offline Policy Optimization27 May
- 2020MOReL : Model-Based Offline Reinforcement Learning12 May
- 2019Behavior Regularized Offline Reinforcement Learning26 Nov
Other references · 18
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2015Trust Region Policy Optimizationcited by 2,580
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
- 2017Constrained Policy Optimizationcited by 983
- 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
- 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
- 2018Behavioral Cloning from Observationcited by 492
- 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
- 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
- 2022Online Decision Transformercited by 196
6 more
- 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
- 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
- 2021A general sample complexity analysis of vanilla policy gradientcited by 76
- 2021Deep reinforcement learning in medical imaging: A literature reviewcited by 34
- 2023PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clippingcited by 17
- 2025Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysiscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
- 2026Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation
- 2026Active Offline-to-Online Reinforcement Learning
- 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
- 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learning
- 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning
- 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
- 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias