mlsanityData to 7 Oct 2026

Policy Expansion for Bridging Offline-to-Online Reinforcement Learning

arXiv 2302.00935 ↗First posted 2 Feb 2023cs.AIcited by 103

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Pre-training with offline data and online fine-tuning using reinforcement learning is a promising strategy for learning control policies by leveraging the best of both worlds in terms of sample efficiency and performance. One natural approach is to initialize the policy for online learning with the one trained offline. In this work, we introduce a policy expansion scheme for this task. After learning the offline policy, we use it as one candidate policy in a policy set. We then expand the policy set with another policy which will be responsible for further learning. The two policies will be composed in an adaptive manner for interacting with the environment. With this approach, the policy previously learned offline is fully retained during online learning, thus mitigating the potential issues such as destroying the useful behaviors of the offline policy in the initial stage of online learning while allowing the offline policy participate in the exploration naturally in an adaptive manner. Moreover, new useful behaviors can potentially be captured by the newly added policy through learning. Experiments are conducted on a number of tasks and the results demonstrate the effectiveness of the proposed approach. Code is available at https://github.com/Haichao-Zhang/PEX

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 52 of its 67 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

52 of 67 references.

Closest ancestors · 13

Same direction, newest first

  1. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories12 Oct
  2. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  3. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  4. 2021Pessimistic Model Selection for Offline Deep Reinforcement Learning29 Nov
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  9. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  10. 2021Representation Matters: Offline Pretraining for Sequential Decision Making11 Feb
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
1 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
27 more
  1. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  2. 2018Do Better ImageNet Models Transfer Better?cited by 560
  3. 2015Policy Distillationcited by 414
  4. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  5. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  6. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  7. 2019Causal Confusion in Imitation Learningcited by 286
  8. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  9. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  10. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  11. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  12. 2022Jump-Start Reinforcement Learningcited by 102
  13. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  14. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  15. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  16. 2020Fighting Copycat Agents in Behavioral Cloning from Observation Historiescited by 70
  17. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  18. 2017Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-tuningcited by 48
  19. 2021Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planningcited by 43
  20. 2020Behavior Priors for Efficient Reinforcement Learningcited by 40
  21. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25
  22. 2021Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learningcited by 21
  23. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learningcited by 18
  24. 2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 18
  25. 2021TAAC: Temporally Abstract Actor-Critic for Continuous Controlcited by 14
  26. 2022Generative Planning for Temporally Coordinated Exploration in Reinforcement Learningcited by 10
  27. 2018A Data-Driven Approach for Autonomous Motion Planning and Control in Off-Road Driving Scenarioscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 28; 2025: 28; 2026: 23.

103 papers.

Sort:

By area:

  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  2. 2024Diffusion Policy Policy Optimizationcited by 255
  3. 2025Reinforcement Learning with Action Chunkingcited by 97
  4. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  5. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  6. 2026Q-learning with Adjoint Matchingcited by 33
  7. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  8. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  9. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  10. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  11. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  12. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
38 more
  1. 2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 18
  2. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  3. 2023Blending Imitation and Reinforcement Learning for Robust Policy Improvementcited by 16
  4. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gapscited by 15
  5. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learningcited by 15
  6. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learningcited by 14
  7. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  8. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  9. 2025Skill Expansion and Composition in Parameter Spacecited by 13
  10. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
  11. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
  12. 2023Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agentscited by 12
  13. 2025Reinforcement Learning via Implicit Imitation Guidancecited by 12
  14. 2024Unsupervised-to-Online Reinforcement Learningcited by 11
  15. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learningcited by 10
  16. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RLcited by 10
  17. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  18. 2025Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generationcited by 9
  19. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  20. 2024Robot Policy Learning with Temporal Optimal Transport Rewardcited by 8
  21. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  22. 2025Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Datacited by 8
  23. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  24. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learningcited by 7
  25. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Biascited by 7
  26. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
  27. 2024DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delayscited by 6
  28. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  29. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Datacited by 5
  30. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RLcited by 5
  31. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learningcited by 4
  32. 2025SLIM: Sim-to-Real Legged Instructive Manipulation via Long-Horizon Visuomotor Learningcited by 4
  33. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learningcited by 4
  34. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptationcited by 4
  35. 2025The Three Regimes of Offline-to-Online Reinforcement Learningcited by 4
  36. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 4
  37. 2023RLSynC: Offline-Online Reinforcement Learning for Synthon Completioncited by 3
  38. 2023Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approachcited by 3

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning
  2. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  3. 2026Active Offline-to-Online Reinforcement Learning
  4. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  5. 2022MOORe: Model-based Offline-to-Online Reinforcement Learning
  6. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  7. 2022A Unified Framework for Alternating Offline Model Training and Policy Learning
  8. 2024Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning