mlsanityData to 14 Oct 2026

ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles

arXiv 2306.06871 ↗First posted 12 Jun 2023cs.LG cs.AI cs.ROcited by 20

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) is a learning paradigm where an agent learns from a fixed dataset of experience. However, learning solely from a static dataset can limit the performance due to the lack of exploration. To overcome it, offline-to-online RL combines offline pre-training with online fine-tuning, which enables the agent to further refine its policy by interacting with the environment in real-time. Despite its benefits, existing offline-to-online RL methods suffer from performance degradation and slow improvement during the online phase. To tackle these challenges, we propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Moreover, to expedite online performance enhancement, we appropriately loosen the pessimism of Q-value estimation and incorporate ensemble-based exploration mechanisms into our framework. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods during online fine-tuning on a range of locomotion and navigation tasks, significantly outperforming existing offline-to-online RL methods.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 52 of its 64 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

52 of 64 references.

Closest ancestors · 23

Same direction, newest first

  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  2. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  3. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning25 Oct
  4. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  5. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  6. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  7. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  8. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RL16 Mar
  9. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  10. 2022MOORe: Model-based Offline-to-Online Reinforcement Learning25 Jan
  11. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  12. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
11 more
  1. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  2. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  5. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOPO: Model-based Offline Policy Optimization27 May
  8. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 29

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2013Auto-Encoding Variational Bayescited by 4,053
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  11. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  12. 2016Deep Exploration via Bootstrapped DQNcited by 527
17 more
  1. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  2. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  3. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  4. 2020First return, then explorecited by 266
  5. 2017Count-Based Exploration with Neural Density Modelscited by 257
  6. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  7. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  8. 2020Model-Based Offline Planningcited by 129
  9. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  10. 2017Bayesian Compression for Deep Learningcited by 116
  11. 2018Episodic Curiosity through Reachabilitycited by 109
  12. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  13. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  14. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
  15. 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
  16. 2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scalecited by 44
  17. 2021Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planningcited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 5; 2026: 4.

20 papers.

Sort:

By area:

  1. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
  2. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guaranteescited by 11
  3. 2024The Evolution of Reinforcement Learning in Quantitative Finance: A Surveycited by 11
  4. 2024Unsupervised-to-Online Reinforcement Learningcited by 11
  5. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  6. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
  7. 2023Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimizationcited by 3
  8. 2024Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Explorationcited by 3
  9. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruptioncited by 3
  10. 2024Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learningcited by 2
  11. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learningcited by 2
  12. 2026RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Rankingcited by 2
8 more
  1. 2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learningcited by 1
  2. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RLcited by 1
  3. 2024SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidancecited by 0
  4. 2025HEAT:History-Enhanced Dual-phase Actor-Critic Algorithm with A Shared Transformercited by 0
  5. 2025Fine-Tuning without Performance Degradationcited by 0
  6. 2026SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfercited by 0
  7. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithmcited by 0
  8. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  2. 2025Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
  3. 2026Active Offline-to-Online Reinforcement Learning
  4. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  5. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
  6. 2025Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only
  7. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
  8. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data