mlsanityData to 14 Oct 2026

Leveraging Offline Data in Online Reinforcement Learning

arXiv 2211.04974 ↗First posted 9 Nov 2022cs.LG cs.AI stat.MLcited by 42

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Two central paradigms have emerged in the reinforcement learning (RL) community: online RL and offline RL. In the online RL setting, the agent has no prior knowledge of the environment, and must interact with it in order to find an ε-optimal policy. In the offline RL setting, the learner instead has access to a fixed dataset to learn from, but is unable to otherwise interact with the environment, and must obtain the best policy it can from this offline data. Practical scenarios often motivate an intermediate setting: if we have some set of offline data and, in addition, may also interact with the environment, how can we best use the offline data to minimize the number of online interactions necessary to learn an ε-optimal policy? In this work, we consider this setting, which we call the textsfFineTuneRL setting, for MDPs with linear structure. We characterize the necessary number of online samples needed in this setting given access to some offline dataset, and develop an algorithm, textscFTPedel, which is provably optimal, up to H factors. We show through an explicit example that combining offline data with online interactions can lead to a provable improvement over either purely offline or purely online RL. Finally, our results illustrate the distinction between verifiable learning, the typical setting considered in online RL, and unverifiable learning, the setting often considered in offline RL, and show that there is a formal separation between these regimes.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 64 of its 91 references that are in the map. First posted on arXiv 9 Nov 2022.

Builds on

64 of 91 references.

Closest ancestors · 18

Same direction, newest first

  1. 2022The Role of Coverage in Online Reinforcement Learning9 Oct
  2. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps25 Mar
  3. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism11 Mar
  4. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  5. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  6. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  8. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
  9. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  10. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  11. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction2 Feb
  12. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
6 more
  1. 2020Batch Value-function Approximation with Only Realizability11 Aug
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020MOPO: Model-based Offline Policy Optimization27 May
  4. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  6. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  7. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  8. 2018Is Q-learning Provably Efficient?cited by 487
  9. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  10. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  11. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  12. 2014On the Complexity of Best Arm Identification in Multi-Armed Bandit Modelscited by 286
34 more
  1. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 234
  2. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 223
  3. 2021The Statistical Complexity of Interactive Decision Makingcited by 218
  4. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 201
  5. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 200
  6. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 195
  7. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 181
  8. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 178
  9. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 176
  10. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  11. 2019Non-Asymptotic Gap-Dependent Regret Bounds for Tabular MDPscited by 105
  12. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  13. 2019Frequentist Regret Bounds for Randomized Least-Squares Value Iterationcited by 101
  14. 2019Sequential Experimental Design for Transductive Linear Banditscited by 91
  15. 2020Logarithmic Regret for Reinforcement Learning with Linear Function Approximationcited by 79
  16. 2020Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functionscited by 71
  17. 2020Q-learning with Logarithmic Regretcited by 60
  18. 2020Provably Efficient Reward-Agnostic Navigation with Linear Value Iterationcited by 56
  19. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 52
  20. 2021Fine-Grained Gap-Dependent Bounds for Tabular MDPs via Adaptive Multi-Step Bootstrapcited by 49
  21. 2022Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processescited by 49
  22. 2021Beyond No Regret: Instance-Dependent PAC Reinforcement Learningcited by 41
  23. 2021First-Order Regret in Reinforcement Learning with Linear Function Approximation: A Robust Estimation Approachcited by 38
  24. 2019The True Sample Complexity of Identifying Good Armscited by 37
  25. 2018Exploration in Structured Reinforcement Learningcited by 34
  26. 2021Beyond Value-Function Gaps: Improved Instance-Dependent Regret Bounds for Episodic Reinforcement Learningcited by 34
  27. 2022Instance-Dependent Near-Optimal Policy Identification in Linear MDPs via Online Experiment Designcited by 33
  28. 2020Online Sparse Reinforcement Learningcited by 25
  29. 2021A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learningcited by 24
  30. 2020Bandits with Partially Observable Confounded Datacited by 22
  31. 2021Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPscited by 19
  32. 2022Optimistic PAC Reinforcement Learning: the Instance-Dependent Viewcited by 17
  33. 2020Towards Tractable Optimism in Model-Based Reinforcement Learningcited by 11
  34. 2020Safety Verification of Model Based Reinforcement Learning Controllerscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 11; 2025: 9; 2026: 8.

42 papers.

Sort:

By area:

  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  3. 2023Synthetic Experience Replaycited by 101
  4. 2022Transfer Learning for Contextual Multi-armed Banditscited by 24
  5. 2023Optimal Exploration for Model-Based RL in Nonlinear Systemscited by 18
  6. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gapscited by 17
  7. 2023Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learningcited by 16
  8. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learningcited by 14
  9. 2023Leveraging Demonstrations to Improve Online Learning: Quality Matterscited by 13
  10. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 11
  11. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RLcited by 10
  12. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
30 more
  1. 2023(Re)2H2O: Autonomous Driving Scenario Generation via Reversely Regularized Hybrid Offline-and-Online Reinforcement Learningcited by 6
  2. 2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6
  3. 2024Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPscited by 6
  4. 2025Balancing optimism and pessimism in offline-to-online learningcited by 6
  5. 2025SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learningcited by 6
  6. 2023Bridging Imitation and Online Reinforcement Learning: An Optimistic Talecited by 5
  7. 2024A Natural Extension To Online Algorithms For Hybrid RL With Limited Coveragecited by 5
  8. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  9. 2023What can online reinforcement learning with function approximation benefit from general coverage conditions?cited by 4
  10. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learningcited by 4
  11. 2025Contextual Online Pricing with (Biased) Offline Datacited by 4
  12. 2023Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approachcited by 3
  13. 2024Hybrid Training for Enhanced Multi-task Generalization in Multi-agent Reinforcement Learningcited by 3
  14. 2025Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysiscited by 3
  15. 2023Warm-Start Actor-Critic: From Approximation Error to Sub-optimality Gapcited by 2
  16. 2024Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneitycited by 2
  17. 2024Leveraging Unlabeled Data Sharing through Kernel Function Approximation in Offline Reinforcement Learningcited by 2
  18. 2025On The Statistical Complexity of Offline Decision-Makingcited by 2
  19. 2025Statistical and Algorithmic Foundations of Reinforcement Learningcited by 2
  20. 2026Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environmentscited by 2
  21. 2023Robust Knowledge Transfer in Tiered Reinforcement Learningcited by 1
  22. 2024Online Residual Learning from Offline Experts for Pedestrian Trackingcited by 1
  23. 2026Direction-Aware Offline-to-Online Learning in Linear Contextual Banditscited by 1
  24. 2025Deconfounded Warm-Start Thompson Sampling with Applications to Precision Medicinecited by 0
  25. 2026Sample-Mean Anchored Thompson Sampling for Offline-to-Online Learning with Distribution Shiftcited by 0
  26. 2026ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimizationcited by 0
  27. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  28. 2026Information-Directed Offline-to-Online Reinforcement Learningcited by 0
  29. 2026Expert Behavior Prior Reinforcement Learningcited by 0
  30. 2026Improving Online Reinforcement Learning via Bidirectional Behavior Prior Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  2. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
  3. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  4. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  5. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
  6. 2021A Minimalist Approach to Offline Reinforcement Learning
  7. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  8. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data