mlsanityData to 7 Oct 2026

Bridging Imitation and Online Reinforcement Learning: An Optimistic Tale

arXiv 2303.11369 ↗First posted 20 Mar 2023cs.LG cs.AIcited by 5

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

In this paper, we address the following problem: Given an offline demonstration dataset from an imperfect expert, what is the best way to leverage it to bootstrap online learning performance in MDPs. We first propose an Informed Posterior Sampling-based RL (iPSRL) algorithm that uses the offline dataset, and information about the expert's behavioral policy used to generate the offline dataset. Its cumulative Bayesian regret goes down to zero exponentially fast in N, the offline dataset size if the expert is competent enough. Since this algorithm is computationally impractical, we then propose the iRLSVI algorithm that can be seen as a combination of the RLSVI algorithm for online RL, and imitation learning. Our empirical results show that the proposed iRLSVI algorithm is able to achieve significant reduction in regret as compared to two baselines: no offline data, and offline dataset but used without information about the generative policy. Our algorithm bridges online RL and imitation learning for the first time.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 52 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

42 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Imitation Learning by Estimating Expertise of Demonstrators2 Feb
  2. 2019Go-Explore: a New Approach for Hard-Exploration Problems30 Jan

Other references · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2016Deep Exploration via Bootstrapped DQNcited by 518
  12. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
28 more
  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  2. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  4. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  5. 2020First return, then explorecited by 256
  6. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  7. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  8. 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
  9. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  10. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  11. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  12. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  13. 2017Deep Exploration via Randomized Value Functionscited by 155
  14. 2020Model-Based Offline Planningcited by 126
  15. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  16. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  17. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  18. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  19. 2017Learning Unknown Markov Decision Processes: A Thompson Sampling Approachcited by 65
  20. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
  21. 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
  22. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  23. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  24. 2022Model-based RL with Optimistic Posterior Sampling: Structural Conditions and Sample Complexitycited by 31
  25. 2022A Provably Efficient Model-Free Posterior Sampling Method for Episodic Reinforcement Learningcited by 30
  26. 2022Regret Bounds for Information-Directed Reinforcement Learningcited by 24
  27. 2022Safe Exploration for Efficient Policy Evaluation and Comparisoncited by 15
  28. 2022Optimistic Posterior Sampling for Reinforcement Learning with Few Samples and Tight Guaranteescited by 12

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 2.

5 papers.

  1. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  2. 2025Best Policy Learning from Trajectory Preference Feedbackcited by 3
  3. 2024Physics-informed Imitative Reinforcement Learning for Real-world Drivingcited by 2
  4. 2024Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneitycited by 1
  5. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithmscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Strong Baseline for Batch Imitation Learning
  2. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  3. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  4. 2022On Efficient Online Imitation Learning via Classification
  5. 2025Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
  6. 2022Improved Policy Optimization for Online Imitation Learning
  7. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learning
  8. 2024Online Bandit Learning with Offline Preference Data for Improved RLHF