mlsanityData to 7 Oct 2026

Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient

arXiv 2210.06718 ↗First posted 13 Oct 2022cs.LGcited by 158

Not on the map. Why?

Cited by 68 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We consider a hybrid reinforcement learning setting (Hybrid RL), in which an agent has access to an offline dataset and the ability to collect experience via real-world online interaction. The framework mitigates the challenges that arise in both pure offline and online RL settings, allowing for the design of simple and highly effective algorithms, in both theory and practice. We demonstrate these advantages by adapting the classical Q learning/iteration algorithm to the hybrid setting, which we call Hybrid Q-Learning or Hy-Q. In our theoretical results, we prove that the algorithm is both computationally and statistically efficient whenever the offline dataset supports a high-quality policy and the environment has bounded bilinear rank. Notably, we require no assumptions on the coverage provided by the initial distribution, in contrast with guarantees for policy gradient/iteration methods. In our experimental results, we show that Hy-Q with neural network function approximation outperforms state-of-the-art online, offline, and hybrid RL baselines on challenging benchmarks, including Montezuma's Revenge.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Builds on 56 of its 82 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

56 of 82 references.

References · 56

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  10. 2015Prioritized Experience Replaycited by 1,070
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
38 more
  1. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  3. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  4. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  5. 2014Convex Optimization: Algorithms and Complexitycited by 569
  6. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  8. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  9. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  10. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  11. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  12. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  13. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  14. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  15. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  16. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  17. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  18. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
  19. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  20. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  21. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  22. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  23. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  24. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  25. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  26. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  27. 2020Batch Value-function Approximation with Only Realizabilitycited by 103
  28. 2022Jump-Start Reinforcement Learningcited by 102
  29. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  30. 2018On Oracle-Efficient PAC RL with Rich Observationscited by 71
  31. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  32. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 69
  33. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 59
  34. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  35. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  36. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
  37. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  38. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 29; 2024: 44; 2025: 47; 2026: 35.

158 papers.

Sort:

By area:

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  2. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  3. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  4. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  5. 2025Flow Q-Learningcited by 162
  6. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  7. 2025Reinforcement Learning with Action Chunkingcited by 97
  8. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  9. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  10. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 72
  11. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  12. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
38 more
  1. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  2. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 57
  3. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  4. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  5. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  6. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 51
  7. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  8. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  9. 2026Q-learning with Adjoint Matchingcited by 33
  10. 2024Hybrid Inverse Reinforcement Learningcited by 31
  11. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  12. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  13. 2022Transfer Learning for Contextual Multi-armed Banditscited by 24
  14. 2023RLIF: Interactive Imitation Learning as Reinforcement Learningcited by 24
  15. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 23
  16. 2024Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flightcited by 22
  17. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  18. 2025Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Modelscited by 19
  19. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  20. 2024Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RLcited by 17
  21. 2023Blending Imitation and Reinforcement Learning for Robust Policy Improvementcited by 16
  22. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  23. 2023Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learningcited by 15
  24. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gapscited by 15
  25. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learningcited by 14
  26. 2025What Matters for Batch Online Reinforcement Learning in Robotics?cited by 14
  27. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  28. 2023Leveraging Demonstrations to Improve Online Learning: Quality Matterscited by 13
  29. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  30. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  31. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
  32. 2023The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithmscited by 12
  33. 2023REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulationcited by 12
  34. 2025Reinforcement Learning via Implicit Imitation Guidancecited by 12
  35. 2025Step-DAD: Semi-Amortized Policy-Based Bayesian Experimental Designcited by 12
  36. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 12
  37. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guaranteescited by 11
  38. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learningcited by 11

50 most cited of 158 All 158 (JSON)