mlsanityData to 14 Oct 2026

PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning

arXiv 2305.15669 ↗First posted 25 May 2023cs.LG cs.AI cs.ROcited by 29

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline-to-online reinforcement learning (RL), by combining the benefits of offline pretraining and online finetuning, promises enhanced sample efficiency and policy performance. However, existing methods, effective as they are, suffer from suboptimal performance, limited adaptability, and unsatisfactory computational efficiency. We propose a novel framework, PROTO, which overcomes the aforementioned limitations by augmenting the standard RL objective with an iteratively evolving regularization term. Performing a trust-region-style update, PROTO yields stable initial finetuning and optimal final performance by gradually evolving the regularization term to relax the constraint strength. By adjusting only a few lines of code, PROTO can bridge any offline policy pretraining and standard off-policy RL finetuning to form a powerful offline-to-online RL pathway, birthing great adaptability to diverse methods. Simple yet elegant, PROTO imposes minimal additional computation and enables highly efficient online finetuning. Extensive experiments demonstrate that PROTO achieves superior performance over SOTA baselines, offering an adaptable and efficient offline-to-online RL framework.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 62 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

51 of 62 references.

Closest ancestors · 25

Same direction, newest first

  1. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions30 Mar
  2. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  3. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  4. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  5. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
  6. 2023Behavior Proximal Policy Optimization22 Feb
  7. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewards3 Feb
  8. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  9. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  10. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning8 Dec
  11. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning25 Oct
  12. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
13 more
  1. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  2. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  3. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  4. 2021False Correlation Reduction for Offline Reinforcement Learning24 Oct
  5. 2021Offline Reinforcement Learning with Soft Behavior Regularization14 Oct
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  13. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2016Layer Normalizationcited by 5,669
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
14 more
  1. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  4. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  5. 2019Challenges of Real-World Reinforcement Learningcited by 291
  6. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  7. 2019Batch Policy Learning under Constraintscited by 244
  8. 2022Online Decision Transformercited by 196
  9. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183
  10. 2022Jump-Start Reinforcement Learningcited by 112
  11. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  12. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 65
  13. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  14. 2017Trust-PCL: An Off-Policy Trust Region Method for Continuous Controlcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 10; 2025: 8; 2026: 6.

29 papers.

Sort:

By area:

  1. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
  2. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
  3. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  4. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  5. 2024A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agentscited by 16
  6. 2026Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulationcited by 11
  7. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  8. 2025Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generationcited by 9
  9. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  10. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  11. 2025Dichotomous Diffusion Policy Optimizationcited by 6
  12. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 5
17 more
  1. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learningcited by 4
  2. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  3. 2025The Three Regimes of Offline-to-Online Reinforcement Learningcited by 4
  4. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
  5. 2025Generative Actor Criticcited by 2
  6. 2023A Fully Data-Driven Approach for Realistic Traffic Signal Control Using Offline Reinforcement Learningcited by 1
  7. 2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learningcited by 1
  8. 2024Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Searchcited by 1
  9. 2023Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Explorationcited by 0
  10. 2025Dual Alignment Maximin Optimization for Offline Model-based RLcited by 0
  11. 2025Fine-Tuning without Performance Degradationcited by 0
  12. 2025Online Pre-Training for Offline-to-Online Reinforcement Learningcited by 0
  13. 2026SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfercited by 0
  14. 2026ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimizationcited by 0
  15. 2026Information-Directed Offline-to-Online Reinforcement Learningcited by 0
  16. 2026Sim2O: Efficient Offline-to-Online MARL via Joint Action Compositioncited by 0
  17. 2026Beyond Safe Execution: Safe Online Reinforcement Learning for Autonomous Driving with Value Calibration and Adaptive Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithm
  2. 2025A Clean Slate for Offline Reinforcement Learning
  3. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning
  4. 2026Active Offline-to-Online Reinforcement Learning
  5. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  6. 2022MOORe: Model-based Offline-to-Online Reinforcement Learning
  7. 2024A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning
  8. 2026Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation