mlsanityData to 7 Oct 2026

Strategically Conservative Q-Learning

arXiv 2406.04534 ↗First posted 6 Jun 2024cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) is a compelling paradigm to extend RL's practical utility by leveraging pre-collected, static datasets, thereby avoiding the limitations associated with collecting online interactions. The major difficulty in offline RL is mitigating the impact of approximation errors when encountering out-of-distribution (OOD) actions; doing so ineffectively will lead to policies that prefer OOD actions, which can lead to unexpected and potentially catastrophic results. Despite the variety of works proposed to address this issue, they tend to excessively suppress the value function in and around OOD regions, resulting in overly pessimistic value estimates. In this paper, we propose a novel framework called Strategically Conservative Q-Learning (SCQ) that distinguishes between OOD data that is easy and hard to estimate, ultimately resulting in less conservative value estimates. Our approach exploits the inherent strengths of neural networks to interpolate, while carefully navigating their limitations in extrapolation, to obtain pessimistic yet still property calibrated value estimates. Theoretical analysis also shows that the value function learned by SCQ is still conservative, but potentially much less so than that of Conservative Q-learning (CQL). Finally, extensive evaluation on the D4RL benchmark tasks shows our proposed method outperforms state-of-the-art methods. Our code is available through urlhttps://github.com/purewater0901/SCQ.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 48 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

35 of 48 references.

Closest ancestors · 18

Same direction, newest first

  1. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL6 Oct
  2. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  4. 2023Anti-Exploration by Random Network Distillation31 Jan
  5. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes28 Nov
  6. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  7. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  8. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
  9. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  10. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
6 more
  1. 2021Offline Reinforcement Learning with Pseudometric Learning2 Mar
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  3. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 17

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  11. 2021Implicit Behavioral Cloningcited by 442
  12. 2019Batch Policy Learning under Constraintscited by 242
5 more
  1. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
  2. 2018Learning to Walk via Deep Reinforcement Learningcited by 153
  3. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  4. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  5. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  2. 2023Contextual Conservative Q-Learning for Offline Reinforcement Learning
  3. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning
  4. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  5. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model
  6. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning
  7. 2023Conservative State Value Estimation for Offline Reinforcement Learning
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble