mlsanityData to 7 Oct 2026

Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning

arXiv 2305.10282 ↗First posted 17 May 2023cs.LG cs.IT math.ITcited by 15

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

This paper studies tabular reinforcement learning (RL) in the hybrid setting, which assumes access to both an offline dataset and online interactions with the unknown environment. A central question boils down to how to efficiently utilize online data collection to strengthen and complement the offline dataset and enable effective policy fine-tuning. Leveraging recent advances in reward-agnostic exploration and model-based offline RL, we design a three-stage hybrid RL algorithm that beats the best of both worlds -- pure offline RL and pure online RL -- in terms of sample complexities. The proposed algorithm does not require any reward information during data collection. Our theory is developed based on a new notion called single-policy partial concentrability, which captures the trade-off between distribution mismatch and miscoverage and guides the interplay between offline and online data.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 56 of its 75 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

56 of 75 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Minimax-Optimal Reward-Agnostic Exploration in Reinforcement Learning14 Apr
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  3. 2022Leveraging Offline Data in Online Reinforcement Learning9 Nov
  4. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  5. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
  6. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  7. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  8. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  9. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  10. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  11. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction2 Feb
  12. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
3 more
  1. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  2. 2020On Reward-Free Reinforcement Learning with Linear Function Approximation19 Jun
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun

Other references · 41

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  3. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  4. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  6. 2014Convex Optimization: Algorithms and Complexitycited by 569
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  8. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  9. 2018Is Q-learning Provably Efficient?cited by 485
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  12. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
29 more
  1. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  2. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  3. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  4. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  5. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  6. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
  7. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  8. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  9. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
  10. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  11. 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 104
  12. 2020Fast active learning for pure exploration in reinforcement learningcited by 92
  13. 2019Provably Efficient Q-Learning with Low Switching Costcited by 77
  14. 2020Adaptive Reward-Free Explorationcited by 77
  15. 2019Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDPcited by 53
  16. 2021Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learningcited by 51
  17. 2022Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processescited by 49
  18. 2021UCB Momentum Q-learning: Correcting the bias without forgettingcited by 45
  19. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 41
  20. 2020Task-agnostic Exploration in Reinforcement Learningcited by 37
  21. 2022A General Framework for Sample-Efficient Function Approximation in Reinforcement Learningcited by 31
  22. 2021Reward-Free Model-Based Reinforcement Learning with Linear Function Approximationcited by 29
  23. 2020Model-Free Reinforcement Learning: from Clipped Pseudo-Regret to Sample Complexitycited by 28
  24. 2021Sample-Efficient Reinforcement Learning Is Feasible for Linearly Realizable MDPs with Limited Revisitingcited by 27
  25. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?cited by 27
  26. 2022On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RLcited by 27
  27. 2022Transfer Learning for Contextual Multi-armed Banditscited by 24
  28. 2022Near-Optimal Deployment Efficiency in Reward-Free Reinforcement Learning with Linear Function Approximationcited by 13
  29. 2022Safe Exploration Incurs Nearly No Additional Sample Complexity for Reward-free RLcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 6; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Datacited by 11
  2. 2024Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Datacited by 8
  3. 2024Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPscited by 6
  4. 2024A Natural Extension To Online Algorithms For Hybrid RL With Limited Coveragecited by 5
  5. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  6. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learningcited by 4
  7. 2025Balancing optimism and pessimism in offline-to-online learningcited by 4
  8. 2025Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysiscited by 3
  9. 2024Multiple-policy Evaluation via Density Estimationcited by 2
  10. 2025On The Statistical Complexity of Offline Decision-Makingcited by 2
  11. 2025Statistical and Algorithmic Foundations of Reinforcement Learningcited by 2
  12. 2025Hybrid Combinatorial Multi-armed Bandits with Probabilistically Triggered Armscited by 2
3 more
  1. 2024Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloningcited by 1
  2. 2025Learning Upper Lower Value Envelopes to Shape Online RL: A Principled Approachcited by 0
  3. 2026Provably Efficient Offline-to-Online Value Adaptation with General Function Approximationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
  2. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  3. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  4. 2025The Three Regimes of Offline-to-Online Reinforcement Learning
  5. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  6. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  7. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
  8. 2025Online Optimization for Offline Safe Reinforcement Learning