Data to 7 Oct 2026

Conservative Q-Learning for Offline Reinforcement Learning

arXiv 2006.04779 ↗First posted 8 Jun 2020cs.LG stat.MLcited by 1,777

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Effectively leveraging large, previously collected datasets in reinforcement learning (RL) is a key challenge for large-scale real-world applications. Offline RL algorithms promise to learn effective policies from previously-collected, static datasets without further interaction. However, in practice, offline RL presents a major challenge, and standard off-policy RL methods can fail due to overestimation of values induced by the distributional shift between the dataset and the learned policy, especially when training on complex and multi-modal data distributions. In this paper, we propose conservative Q-learning (CQL), which aims to address these limitations by learning a conservative Q-function such that the expected value of a policy under this Q-function lower-bounds its true value. We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees. In practice, CQL augments the standard Bellman error objective with a simple Q-value regularizer which is straightforward to implement on top of existing deep Q-learning and actor-critic implementations. On both discrete and continuous control domains, we show that CQL substantially outperforms existing offline RL methods, often learning policies that attain 2-5 times higher final return, especially when learning from complex and multi-modal data distributions.

Key facts. Cited by 1,777 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 64 references that are in the map. First posted on arXiv 8 Jun 2020.

Builds on

42 of 64 references.

Closest ancestors · 4

Same direction, newest first

Other references · 38

2015Deep Residual Learning for Image Recognitioncited by 51,809
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2013Playing Atari with Deep Reinforcement Learningcited by 3,211
2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2018Exploration by Random Network Distillationcited by 966
2017Constrained Policy Optimizationcited by 928
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685

26 more

2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
2016Deep Exploration via Bootstrapped DQNcited by 518
2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
2016Variance-based regularization with convex objectivescited by 153
2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
2017Consistent On-Line Off-Policy Evaluationcited by 51
2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
2014Approximate Policy Iteration Schemes: A Comparisoncited by 39
2018Variational Bayesian Reinforcement Learning with Regret Boundscited by 31
2019Safe Policy Improvement with Soft Baseline Bootstrappingcited by 22
2019Safe Policy Improvement with an Estimated Baseline Policycited by 16
2019Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Samplingcited by 8
2018A General Family of Robust Stochastic Operators for Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 8; 2021: 136; 2022: 218; 2023: 354; 2024: 364; 2025: 356; 2026: 341.

1,777 papers.

Sort:

By area:

2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
2021Implicit Behavioral Cloningcited by 442
2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
2020Is Pessimism Provably Efficient for Offline RL?cited by 362
2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337

38 more

2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
2022Behavior Transformers: Cloning k modes with one stonecited by 299
2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
2023Pretraining Language Models with Human Preferencescited by 245
2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
2022Multi-Game Decision Transformerscited by 212
2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
2022Active Example Selection for In-Context Learningcited by 195
2022Online Decision Transformercited by 194
2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
2022Human-to-Robot Imitation in the Wildcited by 157
2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
2021Offline RL Without Off-Policy Evaluationcited by 150
2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142

50 most cited of 1,777 All 1,777 (JSON)

Similar, not cited

Close in wording, no citation either way