Data to 7 Oct 2026

Mildly Conservative Q-Learning for Offline Reinforcement Learning

arXiv 2206.04745 ↗First posted 9 Jun 2022cs.LG cs.AIcited by 112

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) defines the task of learning from a static logged dataset without continually interacting with the environment. The distribution shift between the learned policy and the behavior policy makes it necessary for the value function to stay conservative such that out-of-distribution (OOD) actions will not be severely overestimated. However, existing approaches, penalizing the unseen actions or regularizing with the behavior policy, are too pessimistic, which suppresses the generalization of the value function and hinders the performance improvement. This paper explores mild but enough conservatism for offline learning while not harming generalization. We propose Mildly Conservative Q-learning (MCQ), where OOD actions are actively trained by assigning them proper pseudo Q values. We theoretically show that MCQ induces a policy that behaves at least as well as the behavior policy and no erroneous overestimation will occur for OOD actions. Experimental results on the D4RL benchmarks demonstrate that MCQ achieves remarkable performance compared with prior work. Furthermore, MCQ shows superior generalization ability when transferring from offline to online, and significantly outperforms baselines. Our code is publicly available at https://github.com/dmksjfl/MCQ.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 81 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

50 of 81 references.

Closest ancestors · 25

Same direction, newest first

13 more

2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
2020Is Pessimism Provably Efficient for Offline RL?30 Dec
2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
2020Critic Regularized Regression26 Jun
2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
2020MOPO: Model-based Offline Policy Optimization27 May
2020MOReL : Model-Based Offline Reinforcement Learning12 May
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 25

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2013Auto-Encoding Variational Bayescited by 3,980
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2017Constrained Policy Optimizationcited by 928
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685

13 more

2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
2017VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learningcited by 199
2019Seeing What a GAN Cannot Generatecited by 148
2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32
2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23
2018MGGAN: Solving Mode Collapse using Manifold Guided Trainingcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 27; 2024: 30; 2025: 29; 2026: 22.

112 papers.

Sort:

By area:

2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
2023Anti-Exploration by Random Network Distillationcited by 49
2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 49
2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 33
2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26
2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26

38 more

2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
2023Supported Trust Region Optimization for Offline Reinforcement Learningcited by 20
2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applicationscited by 17
2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learningcited by 16
2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operatorscited by 14
2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
2025What Matters for Batch Online Reinforcement Learning in Robotics?cited by 14
2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 10
2024Learning from Sparse Offline Datasets via Conservative Density Estimationcited by 9
2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
2024Exclusively Penalized Q-learning for Offline Reinforcement Learningcited by 8
2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
2022State Advantage Weighting for Offline RLcited by 5
2024Transductive Off-policy Proximal Policy Optimizationcited by 5
2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RLcited by 5
2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Datacited by 5
2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Datacited by 5
2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
2026VGAS: Value-Guided Action-Chunk Selection for Few-Shot Vision-Language-Action Adaptationcited by 5
2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learningcited by 4
2023HiBid: A Cross-Channel Constrained Bidding System with Budget Allocation by Hierarchical Offline Deep Reinforcement Learningcited by 4

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way