mlsanityData to 7 Oct 2026

The Three Regimes of Offline-to-Online Reinforcement Learning

arXiv 2510.01460 ↗First posted 1 Oct 2025cs.LG cs.AIcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline-to-online reinforcement learning (RL) has emerged as a practical paradigm that leverages offline datasets for pretraining and online interactions for fine-tuning. However, its empirical behavior is highly inconsistent: design choices of online fine-tuning that work well in one setting can fail completely in another. Guided by the stability--plasticity principle, we propose a framework that can explain this inconsistency: We argue that efficient fine-tuning must preserve the utility of the stronger offline prior, whether that is the pretrained policy or the offline dataset, while maintaining sufficient plasticity. This perspective identifies three regimes of online fine-tuning, each requiring distinct stability properties. We validate this framework through a large-scale empirical study, finding that the results strongly align with its predictions in 45 out of 63 cases, with only 3 opposite mismatches. This work provides a framework for guiding design choices in offline-to-online RL based on the relative performance of the offline dataset and the pretrained policy.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 35 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

28 of 35 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data10 Dec
  2. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning31 May
  3. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning25 May
  4. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  6. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 18

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  5. 2016Progressive Neural Networkscited by 1,332
  6. 2018Experience Replay for Continual Learningcited by 1,074
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  10. 2019Batch Policy Learning under Constraintscited by 242
  11. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  12. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
6 more
  1. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  2. 2022Jump-Start Reinforcement Learningcited by 102
  3. 2023Data Quality in Imitation Learningcited by 72
  4. 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shiftcited by 68
  5. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  6. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policiescited by 2
  2. 2026ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimizationcited by 0
  3. 2026Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priorscited by 0
  4. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation
  2. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  3. 2026Active Offline-to-Online Reinforcement Learning
  4. 2022Leveraging Offline Data in Online Reinforcement Learning
  5. 2025A Clean Slate for Offline Reinforcement Learning
  6. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
  7. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  8. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias