mlsanityData to 14 Oct 2026

A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning

arXiv 2312.07685 ↗First posted 12 Dec 2023cs.LG cs.AIcited by 15

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline-to-online Reinforcement Learning (O2O RL) aims to improve the performance of offline pretrained policy using only a few online samples. Built on offline RL algorithms, most O2O methods focus on the balance between RL objective and pessimism, or the utilization of offline and online samples. In this paper, from a novel perspective, we systematically study the challenges that remain in O2O RL and identify that the reason behind the slow improvement of the performance and the instability of online finetuning lies in the inaccurate Q-value estimation inherited from offline pretraining. Specifically, we demonstrate that the estimation bias and the inaccurate rank of Q-value cause a misleading signal for the policy update, making the standard offline RL algorithms, such as CQL and TD3-BC, ineffective in the online finetuning. Based on this observation, we address the problem of Q-value estimation by two techniques: (1) perturbed value update and (2) increased frequency of Q-value updates. The first technique smooths out biased Q-value estimation with sharp peaks, preventing early-stage policy exploitation of sub-optimal actions. The second one alleviates the estimation bias inherited from offline pretraining by accelerating learning. Extensive experiments on the MuJoco and Adroit environments demonstrate that the proposed method, named SO2, significantly alleviates Q-value estimation issues, and consistently improves the performance against the state-of-the-art methods by up to 83.1%.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 34 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

26 of 34 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  2. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning25 Oct
  3. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  6. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2016OpenAI Gymcited by 1,736
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  10. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  11. 2017Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulationcited by 90
  12. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 84
3 more
  1. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 65
  2. 2023Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priorscited by 36
  3. 2022ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependencycited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 6.

15 papers.

Sort:

By area:

  1. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 63
  2. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
  3. 2026Adaptive Action Chunking via Multi-Chunk Q Value Estimationcited by 4
  4. 2024Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Explorationcited by 3
  5. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
  6. 2025VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Makingcited by 3
  7. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learningcited by 2
  8. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspectivecited by 1
  9. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RLcited by 1
  10. 2026Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learningcited by 1
  11. 2025From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learningcited by 0
  12. 2026SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfercited by 0
3 more
  1. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learningcited by 0
  2. 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learningcited by 0
  3. 2026Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  2. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  3. 2024Strategically Conservative Q-Learning
  4. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning
  5. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  6. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
  7. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  8. 2025Online Pre-Training for Offline-to-Online Reinforcement Learning