mlsanityData to 14 Oct 2026

Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration

arXiv 2310.19805 ↗First posted 7 Oct 2023cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Studying how to fine-tune offline reinforcement learning (RL) pre-trained policy is profoundly significant for enhancing the sample efficiency of RL algorithms. However, directly fine-tuning pre-trained policies often results in sub-optimal performance. This is primarily due to the distribution shift between offline pre-training and online fine-tuning stages. Specifically, the distribution shift limits the acquisition of effective online samples, ultimately impacting the online fine-tuning performance. In order to narrow down the distribution shift between offline and online stages, we proposed Q conditioned state entropy (QCSE) as intrinsic reward. Specifically, QCSE maximizes the state entropy of all samples individually, considering their respective Q values. This approach encourages exploration of low-frequency samples while penalizing high-frequency ones, and implicitly achieves State Marginal Matching (SMM), thereby ensuring optimal performance, solving the asymptotic sub-optimality of constraint-based approaches. Additionally, QCSE can seamlessly integrate into various RL algorithms, enhancing online fine-tuning performance. To validate our claim, we conduct extensive experiments, and observe significant improvements with QCSE (about 13% for CQL and 8% for Cal-QL). Furthermore, we extended experimental tests to other algorithms, affirming the generality of QCSE.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 56 references that are in the map. First posted on arXiv 7 Oct 2023.

Builds on

46 of 56 references.

Closest ancestors · 27

Same direction, newest first

  1. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning22 Jun
  2. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning13 Jun
  3. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning25 May
  4. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions30 Mar
  5. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  7. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
  8. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  9. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning8 Dec
  10. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  11. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  12. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
15 more
  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  7. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  9. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  13. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  14. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  15. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2018Exploration by Random Network Distillationcited by 993
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  11. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
7 more
  1. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  2. 2022Online Decision Transformercited by 196
  3. 2019Efficient Exploration via State Marginal Matchingcited by 188
  4. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  5. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  6. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20
  7. 2021Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learningcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only
  2. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  3. 2026Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
  4. 2026Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?
  5. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
  6. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  7. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
  8. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning