mlsanityData to 7 Oct 2026

A Natural Extension To Online Algorithms For Hybrid RL With Limited Coverage

arXiv 2403.09701 ↗First posted 7 Mar 2024cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Hybrid Reinforcement Learning (RL), leveraging both online and offline data, has garnered recent interest, yet research on its provable benefits remains sparse. Additionally, many existing hybrid RL algorithms (Song et al., 2023; Nakamoto et al., 2023; Amortila et al., 2024) impose coverage assumptions on the offline dataset, but we show that this is unnecessary. A well-designed online algorithm should "fill in the gaps" in the offline dataset, exploring states and actions that the behavior policy did not explore. Unlike previous approaches that focus on estimating the offline data distribution to guide online exploration (Li et al., 2023b), we show that a natural extension to standard optimistic online algorithms -- warm-starting them by including the offline dataset in the experience replay buffer -- achieves similar provable gains from hybrid data even when the offline dataset does not have single-policy concentrability. We accomplish this by partitioning the state-action space into two, bounding the regret on each partition through an offline and an online complexity measure, and showing that the regret of this hybrid RL algorithm can be characterized by the best partition -- despite the algorithm not knowing the partition itself. As an example, we propose DISC-GOLF, a modification of an existing optimistic online algorithm with general function approximation called GOLF used in Jin et al. (2021); Xie et al. (2022a), and show that it demonstrates provable gains over both online-only and offline-only reinforcement learning, with competitive bounds when specialized to the tabular, linear and block MDP cases. Numerical simulations further validate our theory that hybrid data facilitates more efficient exploration, supporting the potential of hybrid RL in various scenarios.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 34 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

32 of 34 references.

References · 32

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  7. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  8. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  9. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
  10. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  11. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  12. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
20 more
  1. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  2. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  3. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  4. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  5. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  6. 2021Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysiscited by 89
  7. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  8. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  9. 2022The Role of Coverage in Online Reinforcement Learningcited by 82
  10. 2020Provably Good Batch Reinforcement Learning Without Great Explorationcited by 77
  11. 2018On Oracle-Efficient PAC RL with Rich Observationscited by 71
  12. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  13. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  14. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  15. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
  16. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  17. 2023Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learningcited by 15
  18. 2024Harnessing Density Ratios for Online Reinforcement Learningcited by 15
  19. 2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
  20. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 1.

5 papers.

  1. 2024Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPscited by 6
  2. 2024RL in Latent MDPs is Tractable: Online Guarantees via Off-Policy Evaluationcited by 5
  3. 2025Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysiscited by 3
  4. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
  5. 2026Sample-Mean Anchored Thompson Sampling for Offline-to-Online Learning with Distribution Shiftcited by 0