Online Pre-Training for Offline-to-Online Reinforcement Learning
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Offline-to-online reinforcement learning (RL) aims to integrate the complementary strengths of offline and online RL by pre-training an agent offline and subsequently fine-tuning it through online interactions. However, recent studies reveal that offline pre-trained agents often underperform during online fine-tuning due to inaccurate value estimation caused by distribution shift, with random initialization proving more effective in certain cases. In this work, we propose a novel method, Online Pre-Training for Offline-to-Online RL (OPT), explicitly designed to address the issue of inaccurate value estimation in offline pre-trained agents. OPT introduces a new learning phase, Online Pre-Training, which allows the training of a new value function tailored specifically for effective online fine-tuning. Implementation of OPT on TD3 and SPOT demonstrates an average 30% improvement in performance across a wide range of D4RL environments, including MuJoCo, Antmaze, and Adroit.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 10 of its 35 references that are in the map. First posted on arXiv 11 Jul 2025.
Builds on
10 of 35 references.
Closest ancestors · 8
Same direction, newest first
- 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning31 May
- 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization6 Nov
- 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning25 May
- 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
- 2019Behavior Regularized Offline Reinforcement Learning26 Nov
Other references · 2
- 2016Layer Normalizationcited by 5,579
- 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
- 2025Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only
- 2026Active Offline-to-Online Reinforcement Learning
- 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
- 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
- 2025The Three Regimes of Offline-to-Online Reinforcement Learning
- 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning