mlsanityData to 14 Oct 2026

Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning

arXiv 2310.03718 ↗First posted 5 Oct 2023cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Safe reinforcement learning (RL) focuses on training reward-maximizing agents subject to pre-defined safety constraints. Yet, learning versatile safe policies that can adapt to varying safety constraint requirements during deployment without retraining remains a largely unexplored and challenging area. In this work, we formulate the versatile safe RL problem and consider two primary requirements: training efficiency and zero-shot adaptation capability. To address them, we introduce the Conditioned Constrained Policy Optimization (CCPO) framework, consisting of two key modules: (1) Versatile Value Estimation (VVE) for approximating value functions under unseen threshold conditions, and (2) Conditioned Variational Inference (CVI) for encoding arbitrary constraint thresholds during policy optimization. Our extensive experiments demonstrate that CCPO outperforms the baselines in terms of safety and task performance while preserving zero-shot adaptation capabilities to different constraint thresholds data-efficiently. This makes our approach suitable for real-world dynamic applications.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 48 of its 63 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

48 of 63 references.

References · 48

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  3. 2017Constrained Policy Optimizationcited by 983
  4. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  5. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  6. 2017Safe Reinforcement Learning via Shieldingcited by 449
  7. 2018Reward Constrained Policy Optimizationcited by 417
  8. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  9. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  10. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  11. 2018Maximum a Posteriori Policy Optimisationcited by 328
  12. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 265
36 more
  1. 2018Safe Exploration in Continuous Action Spacescited by 247
  2. 2020Projection-Based Constrained Policy Optimizationcited by 203
  3. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  4. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 175
  5. 2017Modulating early visual processing by languagecited by 167
  6. 2020Conservative Safety Critics for Explorationcited by 113
  7. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 103
  8. 2019Reward-Conditioned Policiescited by 102
  9. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  10. 2021Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificatescited by 74
  11. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 74
  12. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  13. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  14. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 69
  15. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 62
  16. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 59
  17. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 57
  18. 2022Constrained Policy Optimization via Bayesian World Modelscited by 57
  19. 2019Safe Reinforcement Learning on Autonomous Vehiclescited by 50
  20. 2020A Boolean Task Algebra for Reinforcement Learningcited by 48
  21. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 39
  22. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  23. 2021Multi-Agent Constrained Policy Optimisationcited by 35
  24. 2022Safe Reinforcement Learning for Legged Locomotioncited by 35
  25. 2021Context-Aware Safe Reinforcement Learning for Non-Stationary Environmentscited by 33
  26. 2018Relative Entropy Regularized Policy Iterationcited by 32
  27. 2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 30
  28. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 28
  29. 2021The Logical Options Frameworkcited by 25
  30. 2023Provably Efficient Model-Free Algorithms for Non-stationary CMDPscited by 21
  31. 2020Universal Successor Features for Transfer Reinforcement Learningcited by 20
  32. 2015Attend, Adapt and Transfer: Attentive Deep Architecture for Adaptive Transfer from multiple sources in the same domaincited by 19
  33. 2019Safe Control Algorithms Using Energy Functions: A Unified Framework, Benchmark, and New Directionscited by 17
  34. 2022Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraintscited by 10
  35. 2020Constrained Model-based Reinforcement Learning with Robust Cross-Entropy Methodcited by 7
  36. 2022SAAC: Safe Reinforcement Learning as an Adversarial Game of Actor-Criticscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 7; 2026: 2.

18 papers.

Sort:

By area:

  1. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 57
  2. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 53
  3. 2023Gradient Shaping for Multi-Constraint Safe Reinforcement Learningcited by 10
  4. 2024Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identificationscited by 10
  5. 2024Generalize by Touching: Tactile Ensemble Skill Transfer for Robotic Furniture Assemblycited by 7
  6. 2025AnySafe: Adapting Latent Safety Filters at Runtime via Safety Constraint Parameterization in the Latent Spacecited by 7
  7. 2024An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learningcited by 5
  8. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  9. 2024Feasibility Consistent Representation Learning for Safe Reinforcement Learningcited by 3
  10. 2025QuietPaw: Learning Quadrupedal Locomotion with Versatile Noise Preference Alignmentcited by 2
  11. 2025A Provable Approach for End-to-End Safe Reinforcement Learningcited by 2
  12. 2024TRAM: Test-Time Risk Adaptation with Mixture of Agentscited by 1
6 more
  1. 2026Constraint-Grounded Reinforcement Learning for Variable Impedance Control in Contact-Rich Robotic Insertioncited by 1
  2. 2024mathrmE2CFD: Towards Effective and Efficient Cost Function Design for Safe Reinforcement Learning via Large Language Modelcited by 0
  3. 2025TLXML: Task-Level Explanation of Meta-Learning via Influence Functionscited by 0
  4. 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Modelscited by 0
  5. 2025Constraint-Aware Reinforcement Learning via Adaptive Action Scalingcited by 0
  6. 2026MC-CPO: Mastery-Conditioned Constrained Policy Optimization for Pedagogically Safe Intelligent Tutoring Systemscited by 0