mlsanityData to 14 Oct 2026

Balancing optimism and pessimism in offline-to-online learning

arXiv 2502.08259 ↗First posted 12 Feb 2025cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We consider what we call the offline-to-online learning setting, focusing on stochastic finite-armed bandit problems. In offline-to-online learning, a learner starts with offline data collected from interactions with an unknown environment in a way that is not under the learner's control. Given this data, the learner begins interacting with the environment, gradually improving its initial strategy as it collects more data to maximize its total reward. The learner in this setting faces a fundamental dilemma: if the policy is deployed for only a short period, a suitable strategy (in a number of senses) is the Lower Confidence Bound (LCB) algorithm, which is based on pessimism. LCB can effectively compete with any policy that is sufficiently "covered" by the offline data. However, for longer time horizons, a preferred strategy is the Upper Confidence Bound (UCB) algorithm, which is based on optimism. Over time, UCB converges to the performance of the optimal policy at a rate that is nearly the best possible among all online algorithms. In offline-to-online learning, however, UCB initially explores excessively, leading to worse short-term performance compared to LCB. This suggests that a learner not in control of how long its policy will be in use should start with LCB for short horizons and gradually transition to a UCB-like strategy as more rounds are played. This article explores how and why this transition should occur. Our main result shows that our new algorithm performs nearly as well as the better of LCB and UCB at any point in time. The core idea behind our algorithm is broadly applicable, and we anticipate that our results will extend beyond the multi-armed bandit setting.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 51 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

28 of 51 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees14 Nov
  2. 2023Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning17 May
  3. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  4. 2022Leveraging Offline Data in Online Reinforcement Learning9 Nov
  5. 2022ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data8 Nov
  6. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  7. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
  8. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  11. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  12. 2020MOPO: Model-based Offline Policy Optimization27 May
2 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  2. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 14

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 676
  3. 2017Deep & Cross Network for Ad Click Predictionscited by 393
  4. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  5. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  6. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  7. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  8. 2016Conservative Banditscited by 59
  9. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  10. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28
  11. 2022Pessimism for Offline Linear Contextual Bandits using ℓp Confidence Setscited by 18
  12. 2019Online Pricing with Offline Data: Phase Transition and Inverse Square Lawcited by 13
2 more
  1. 2024Online Bandits with (Biased) Offline Data: Adaptive Learning under Distribution Mismatchcited by 10
  2. 2022Data-pooling Reinforcement Learning for Personalized Healthcare Interventioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Hybrid Combinatorial Multi-armed Bandits with Probabilistically Triggered Armscited by 2
  2. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  3. 2026Offline-Online Reinforcement Learning for Linear Mixture MDPscited by 0
  4. 2026Sample-Mean Anchored Thompson Sampling for Offline-to-Online Learning with Distribution Shiftcited by 0
  5. 2026On the Benefits of Free Exploration for Regret Minimization in Multi-Armed Banditscited by 0
  6. 2026Offline-to-Online Learning in Linear Banditscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Offline Local Search for Online Stochastic Bandits
  2. 2023Survival Instinct in Offline Reinforcement Learning
  3. 2024Is Offline Decision Making Possible with Only Few Samples? Reliable Decisions in Data-Starved Bandits via Trust Region Enhancement
  4. 2024Data-Driven Upper Confidence Bounds with Near-Optimal Regret for Heavy-Tailed Bandits
  5. 2026Gaussian-Width Bounds for Offline-to-Online Linear Bandits
  6. 2020Is Pessimism Provably Efficient for Offline RL?
  7. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  8. 2026Information-Directed Offline-to-Online Reinforcement Learning