mlsanityData to 7 Oct 2026

Learning Safety Constraints from Demonstrations with Unknown Rewards

arXiv 2305.16147 ↗First posted 25 May 2023cs.LG cs.AI stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

We propose Convex Constraint Learning for Reinforcement Learning (CoCoRL), a novel approach for inferring shared constraints in a Constrained Markov Decision Process (CMDP) from a set of safe demonstrations with possibly different reward functions. While previous work is limited to demonstrations with known rewards or fully known environment dynamics, CoCoRL can learn constraints from demonstrations with different unknown rewards without knowledge of the environment dynamics. CoCoRL constructs a convex safe set based on demonstrations, which provably guarantees safety even for potentially sub-optimal (but safe) demonstrations. For near-optimal demonstrations, CoCoRL converges to the true safe set with no policy regret. We evaluate CoCoRL in gridworld environments and a driving simulation with multiple constraints. CoCoRL learns constraints that lead to safe driving behavior. Importantly, we can safely transfer the learned constraints to different tasks and environments. In contrast, alternative methods based on Inverse Reinforcement Learning (IRL) often exhibit poor performance and learn unsafe policies.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 20 of its 57 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

20 of 57 references.

References · 20

  1. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  2. 2018The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systemscited by 276
  3. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  4. 2013One-Class Classification: Taxonomy of Study and Review of Techniquescited by 68
  5. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variablescited by 50
  6. 2020Inverse Constrained Reinforcement Learningcited by 45
  7. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learningcited by 36
  8. 2019Maximum Likelihood Constraint Inference for Inverse Reinforcement Learningcited by 33
  9. 2022Benchmarking Constraint Inference in Inverse Reinforcement Learningcited by 18
  10. 2022Active Learning with Safety Constraintscited by 16
  11. 2021Maximum Likelihood Constraint Inference from Stochastic Demonstrationscited by 15
  12. 2022Learning Soft Constraints From Constrained Expert Demonstrationscited by 13
8 more
  1. 2020Learning Constraints from Locally-Optimal Demonstrations under Cost Function Uncertaintycited by 12
  2. 2022Interactively Learning Preference Constraints in Linear Banditscited by 11
  3. 2019Approximate Robust Control of Uncertain Dynamical Systemscited by 9
  4. 2021Making Human-Like Trade-offs in Constrained Environments by Learning from Demonstrationscited by 7
  5. 2023Maximum Causal Entropy Inverse Constrained Reinforcement Learningcited by 7
  6. 2021Meta-Adversarial Inverse Reinforcement Learning for Decision-making Taskscited by 6
  7. 2021Discretizing Dynamics for Maximum Likelihood Constraint Inferencecited by 3
  8. 2021How To Not Drive: Learning Driving Constraints from Demonstrationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 6; 2026: 5.

19 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2025Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysiscited by 44
  3. 2023Learning Shared Safety Constraints from Multi-task Demonstrationscited by 15
  4. 2025Learning Safety Constraints for Large Language Modelscited by 13
  5. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  6. 2023Directional Optimism for Safe Linear Banditscited by 8
  7. 2025Modeling Others' Minds as Codecited by 6
  8. 2026Safe Exploration via Policy Priorscited by 6
  9. 2025Your Learned Constraint is Secretly a Backward Reachable Tubecited by 2
  10. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  11. 2023Learning Safety Constraints From Demonstration Using One-Class Decision Treescited by 0
  12. 2024Multi Task Inverse Reinforcement Learning for Common Sense Rewardcited by 0
7 more
  1. 2024Learning Soft Driving Constraints from Vectorized Scene Embeddings while Imitating Expert Trajectoriescited by 0
  2. 2025DIAL: Distribution-Informed Adaptive Learning of Multi-Task Constraints for Safety-Critical Systemscited by 0
  3. 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Modelscited by 0
  4. 2026Demonstration-Guided Observation Attacks on Black-Box Safe Reinforcement Learning Controllers for Robotic Systemscited by 0
  5. 2026BarrierSteer: LLM Safety via Learning Barrier Steeringcited by 0
  6. 2026Learning to maintain safety through expert demonstrations in settings with unknown constraints: A Q-learning perspectivecited by 0
  7. 2026Failing Gracefully: Mitigating Impact of Inevitable Robot Failurescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Safe CoR: A Dual-Expert Approach to Integrating Imitation Learning and Safe Reinforcement Learning Using Constraint Rewards
  2. 2018Learning Constraints from Demonstrations
  3. 2026Learning Unknown Constraints without Unsafe Data via Optimality and Counterfactual Regularization
  4. 2025Learning safe, constrained policies via imitation learning: Connection to Probabilistic Inference and a Naive Algorithm
  5. 2026OSIL: Learning Offline Safe Imitation Policies with Safety Inferred from Non-preferred Trajectories
  6. 2024Safe Exploration Using Bayesian World Models and Log-Barrier Optimization
  7. 2023Inverse Constraint Learning and Generalization by Transferable Reward Decomposition ↗
  8. 2026Safe Reinforcement Learning with Preference-based Constraint Inference