mlsanityData to 7 Oct 2026

Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approach

arXiv 2310.11531 ↗First posted 17 Oct 2023cs.LG cs.AI cs.SYcited by 3

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the problem of efficient online reinforcement learning in the infinite horizon setting when there is an offline dataset to start with. We assume that the offline dataset is generated by an expert but with unknown level of competence, i.e., it is not perfect and not necessarily using the optimal policy. We show that if the learning agent models the behavioral policy (parameterized by a competence parameter) used by the expert, it can do substantially better in terms of minimizing cumulative regret, than if it doesn't do that. We establish an upper bound on regret of the exact informed PSRL algorithm that scales as tildeO(√T). This requires a novel prior-dependent regret analysis of Bayesian online learning algorithms for the infinite horizon setting. We then propose the Informed RLSVI algorithm to efficiently approximate the iPSRL algorithm.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

25 of 42 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  10. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  11. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  12. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
13 more
  1. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  2. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  3. 2020Model-Based Offline Planningcited by 126
  4. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  5. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  6. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
  7. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  8. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  9. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learningcited by 40
  10. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  11. 2022Model-based RL with Optimistic Posterior Sampling: Structural Conditions and Sample Complexitycited by 31
  12. 2022Safe Exploration for Efficient Policy Evaluation and Comparisoncited by 15
  13. 2023Leveraging Demonstrations to Improve Online Learning: Quality Matterscited by 13

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025On The Statistical Complexity of Offline Decision-Makingcited by 2
  2. 2025Fine-tuning Behavioral Cloning Policies with Preference-Based Reinforcement Learningcited by 1
  3. 2025Deconfounded Warm-Start Thompson Sampling with Applications to Precision Medicinecited by 0