mlsanityData to 7 Oct 2026

Constraints Penalized Q-learning for Safe Offline Reinforcement Learning

arXiv 2107.09003 ↗First posted 19 Jul 2021cs.LG cs.AIcited by 90

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

We study the problem of safe offline reinforcement learning (RL), the goal is to learn a policy that maximizes long-term reward while satisfying safety constraints given only offline data, without further interaction with the environment. This problem is more appealing for real world RL applications, in which data collection is costly or dangerous. Enforcing constraint satisfaction is non-trivial, especially in offline settings, as there is a potential large discrepancy between the policy distribution and the data distribution, causing errors in estimating the value of safety constraints. We show that na"ive approaches that combine techniques from safe RL and offline RL can only learn sub-optimal solutions. We thus develop a simple yet effective algorithm, Constraints Penalized Q-Learning (CPQ), to solve the problem. Our method admits the use of data generated by mixed behavior policies. We present a theoretical analysis and demonstrate empirically that our approach can learn robustly across a variety of benchmark control tasks, outperforming several baselines.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 39 references that are in the map. First posted on arXiv 19 Jul 2021.

Builds on

27 of 39 references.

References · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2017Constrained Policy Optimizationcited by 928
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
15 more
  1. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  4. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 451
  5. 2018Reward Constrained Policy Optimizationcited by 397
  6. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  7. 2019Batch Policy Learning under Constraintscited by 242
  8. 2018Safe Exploration in Continuous Action Spacescited by 237
  9. 2020Model-Based Offline Planningcited by 126
  10. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 103
  11. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 61
  12. 2019Reinforcement Learning with Convex Constraintscited by 57
  13. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 47
  14. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  15. 2021Model-Based Offline Planning with Trajectory Pruningcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 17; 2024: 19; 2025: 20; 2026: 22.

90 papers.

Sort:

By area:

  1. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  2. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  3. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  4. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  5. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  6. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 59
  7. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  8. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  9. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  10. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
  11. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  12. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
38 more
  1. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  2. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  3. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26
  4. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  5. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  6. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  7. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  8. 2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learningcited by 16
  9. 2023Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Drivingcited by 16
  10. 2024Off-Policy Primal-Dual Safe Reinforcement Learningcited by 13
  11. 2025Skill Expansion and Composition in Parameter Spacecited by 13
  12. 2023RiskBench: A Scenario-based Benchmark for Risk Identificationcited by 12
  13. 2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
  14. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  15. 2023A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learningcited by 9
  16. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  17. 2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
  18. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learningcited by 8
  19. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  20. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learningcited by 7
  21. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorizationcited by 7
  22. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learningcited by 7
  23. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  24. 2022Stabilizing Q-learning with Linear Architectures for Provably Efficient Learningcited by 5
  25. 2022A Near-Optimal Primal-Dual Method for Off-Policy Learning in CMDPcited by 5
  26. 2024Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policycited by 5
  27. 2025V-OCBF: Learning Safety Filters from Offline Data via Value-Guided Offline Control Barrier Functionscited by 5
  28. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learningcited by 4
  29. 2025Data Center Cooling System Optimization Using Offline Reinforcement Learningcited by 4
  30. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  31. 2025Learning Conservative Neural Control Barrier Functions from Offline Datacited by 4
  32. 2022Safe Reinforcement Learning with Contrastive Risk Predictioncited by 3
  33. 2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
  34. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
  35. 2025From Uncertain to Safe: Conformal Adaptation of Diffusion Models for Safe PDE Controlcited by 3
  36. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertaintycited by 3
  37. 2026Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policiescited by 3
  38. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentationcited by 2

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation
  2. 2024Offline Safe Reinforcement Learning Using Trajectory Classification
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
  4. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  6. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  7. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
  8. 2021The Least Restriction for Offline Reinforcement Learning