mlsanityData to 14 Oct 2026

Bayesian Design Principles for Offline-to-Online Reinforcement Learning

arXiv 2405.20984 ↗First posted 31 May 2024cs.LGcited by 14

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) is crucial for real-world applications where exploration can be costly or unsafe. However, offline learned policies are often suboptimal, and further online fine-tuning is required. In this paper, we tackle the fundamental dilemma of offline-to-online fine-tuning: if the agent remains pessimistic, it may fail to learn a better policy, while if it becomes optimistic directly, performance may suffer from a sudden drop. We show that Bayesian design principles are crucial in solving such a dilemma. Instead of adopting optimistic or pessimistic policies, the agent should act in a way that matches its belief in optimal policies. Such a probability-matching agent can avoid a sudden performance drop while still being guaranteed to find the optimal policy. Based on our theoretical findings, we introduce a novel algorithm that outperforms existing methods on various benchmarks, demonstrating the efficacy of our approach. Overall, the proposed approach provides a new perspective on offline-to-online RL that has the potential to enable more effective learning from offline data.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 46 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

40 of 46 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  2. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  3. 2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discovery2 Dec
  4. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes28 Nov
  5. 2022Leveraging Offline Data in Online Reinforcement Learning9 Nov
  6. 2022The Role of Coverage in Online Reinforcement Learning9 Oct
  7. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  8. 2022On the Role of Discount Factor in Offline Reinforcement Learning7 Jun
  9. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  10. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  11. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
7 more
  1. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  4. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  6. 2016Deep Exploration via Bootstrapped DQNcited by 527
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  8. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  9. 2013Learning to Optimize Via Posterior Samplingcited by 256
  10. 2019Provably Efficient Exploration in Policy Optimizationcited by 205
  11. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 201
  12. 2022Online Decision Transformercited by 196
9 more
  1. 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
  2. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  3. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 117
  4. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 79
  5. 2017Learning Unknown Markov Decision Processes: A Thompson Sampling Approachcited by 65
  6. 2019Information-Theoretic Confidence Bounds for Reinforcement Learningcited by 51
  7. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 46
  8. 2022A Provably Efficient Model-Free Posterior Sampling Method for Episodic Reinforcement Learningcited by 31
  9. 2023Bayesian Design Principles for Frequentist Sequential Learningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Datacited by 5
  2. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  3. 2025The Three Regimes of Offline-to-Online Reinforcement Learningcited by 4
  4. 2024Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Explorationcited by 3
  5. 2024Leveraging Unlabeled Data Sharing through Kernel Function Approximation in Offline Reinforcement Learningcited by 2
  6. 2025Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximizationcited by 2
  7. 2025Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Networkcited by 0
  8. 2025Online Pre-Training for Offline-to-Online Reinforcement Learningcited by 0
  9. 2025From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learningcited by 0
  10. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  11. 2026Information-Directed Offline-to-Online Reinforcement Learningcited by 0
  12. 2026Sim2O: Efficient Offline-to-Online MARL via Joint Action Compositioncited by 0
2 more
  1. 2026Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priorscited by 0
  2. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  2. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  3. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning
  4. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  5. 2025Fully Offline Reinforcement Learning
  6. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  7. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  8. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief