mlsanityData to 7 Oct 2026

Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation

arXiv 2604.13966 ↗First posted 15 Apr 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We study value adaptation in offline-to-online reinforcement learning under general function approximation. Starting from an imperfect offline pretrained Q-function, the learner aims to adapt it to the target environment using only a limited amount of online interaction. We first characterize the difficulty of this setting by establishing a minimax lower bound, showing that even when the pretrained Q-function is close to optimal Q^star, online adaptation can be no more efficient than pure online RL on certain hard instances. On the positive side, under a novel structural condition on the offline-pretrained value functions, we propose O2O-LSVI, an adaptation algorithm with problem-dependent sample complexity that provably improves over pure online RL. Finally, we complement our theory with neural-network experiments that demonstrate the practical effectiveness of the proposed method.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 48 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

38 of 48 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation9 Aug
  2. 2024Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPs8 Aug
  3. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees14 Nov
  4. 2023Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning17 May
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  6. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  7. 2022An Empirical Study of Implicit Regularization in Deep Offline RL5 Jul
  8. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism11 Mar
  9. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  10. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
  11. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  12. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
4 more
  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  4. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  5. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  6. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  7. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  8. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  9. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  10. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  11. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  12. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
10 more
  1. 2020Logarithmic Regret for Reinforcement Learning with Linear Function Approximationcited by 79
  2. 2022Nearly Minimax Optimal Reinforcement Learning for Linear Markov Decision Processescited by 57
  3. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 45
  4. 2022VOQL: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximationcited by 44
  5. 2020Nearly Minimax Optimal Reinforcement Learning for Discounted MDPscited by 42
  6. 2022A General Framework for Sample-Efficient Function Approximation in Reinforcement Learningcited by 31
  7. 2021Reward-Free Model-Based Reinforcement Learning with Linear Function Approximationcited by 29
  8. 2023A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximationcited by 14
  9. 2024Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Datacited by 8
  10. 2024Achieving Constant Regret in Linear Markov Decision Processescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning
  2. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  3. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  4. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  5. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
  6. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning
  7. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  8. 2022MOORe: Model-based Offline-to-Online Reinforcement Learning