mlsanityData to 14 Oct 2026

Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis

arXiv 2505.13768 ↗First posted 19 May 2025cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

This paper investigates a hybrid learning framework for reinforcement learning (RL) in which the agent can leverage both an offline dataset and online interactions to learn the optimal policy. We present a unified algorithm and analysis and show that augmenting confidence-based online RL algorithms with the offline dataset outperforms any pure online or offline algorithm alone and achieves state-of-the-art results under two learning metrics, i.e., sub-optimality gap and online learning regret. Specifically, we show that our algorithm achieves a sub-optimality gap tildeO(√1/(N0/mathttC(π^*|ρ)+N1) ), where mathttC(π^*|ρ) is a new concentrability coefficient, N0 and N1 are the numbers of offline and online samples, respectively. For regret minimization, we show that it achieves a constant tildeO( √N1/(N0/mathttC(π-|ρ)+N1) ) speed-up compared to pure online learning, where mathttC(π^-|ρ) is the concentrability coefficient over all sub-optimal policies. Our results also reveal an interesting separation on the desired coverage properties of the offline dataset for sub-optimality gap minimization and regret minimization. We further validate our theoretical findings in several experiments in special RL models such as linear contextual bandits and Markov decision processes (MDPs).

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 40 of its 68 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

40 of 68 references.

References · 40

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  2. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  4. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  5. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  6. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  7. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  8. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  9. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  10. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  11. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 223
  12. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
28 more
  1. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  2. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  3. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  4. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 129
  5. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  6. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 110
  7. 2014Best-Arm Identification in Linear Banditscited by 100
  8. 2020Fast active learning for pure exploration in reinforcement learningcited by 92
  9. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 83
  10. 2020Optimal Best-arm Identification in Linear Banditscited by 71
  11. 2020Stochastic Linear Bandits Robust to Adversarial Attackscited by 65
  12. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 55
  13. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  14. 2022Instance-Dependent Near-Optimal Policy Identification in Linear MDPs via Online Experiment Designcited by 33
  15. 2021Minimax Optimal Fixed-Budget Best Arm Identification in Linear Banditscited by 32
  16. 2021Offline Neural Contextual Bandits: Pessimism, Optimization and Generalizationcited by 25
  17. 2020The Elliptical Potential Lemma Revisitedcited by 23
  18. 2019Warm-starting Contextual Bandits: Robustly Combining Supervised and Bandit Feedbackcited by 21
  19. 2022Pessimism for Offline Linear Contextual Bandits using ℓp Confidence Setscited by 18
  20. 2023Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learningcited by 16
  21. 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Banditscited by 15
  22. 2023Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasetscited by 12
  23. 2020Experimental Design for Regret Minimization in Linear Banditscited by 11
  24. 2020Offline Contextual Bandits with Overparameterized Modelscited by 10
  25. 2024Online Bandits with (Biased) Offline Data: Adaptive Learning under Distribution Mismatchcited by 10
  26. 2023Optimal Best-Arm Identification in Bandits with Access to Offline Datacited by 8
  27. 2024Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPscited by 6
  28. 2024A Natural Extension To Online Algorithms For Hybrid RL With Limited Coveragecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Learning Upper Lower Value Envelopes to Shape Online RL: A Principled Approachcited by 0
  2. 2026Sample-Mean Anchored Thompson Sampling for Offline-to-Online Learning with Distribution Shiftcited by 0
  3. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithmcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Minimalist Approach to Offline Reinforcement Learning
  2. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning
  3. 2025Online Optimization for Offline Safe Reinforcement Learning
  4. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  5. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  6. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  7. 2020MOReL : Model-Based Offline Reinforcement Learning
  8. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions