mlsanityData to 7 Oct 2026

Learning Soft Constraints From Constrained Expert Demonstrations

arXiv 2206.01311 ↗First posted 2 Jun 2022cs.LGcited by 13

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse reinforcement learning (IRL) methods assume that the expert data is generated by an agent optimizing some reward function. However, in many settings, the agent may optimize a reward function subject to some constraints, where the constraints induce behaviors that may be otherwise difficult to express with just a reward function. We consider the setting where the reward function is given, and the constraints are unknown, and propose a method that is able to recover these constraints satisfactorily from the expert data. While previous work has focused on recovering hard constraints, our method can recover cumulative soft constraints that the agent satisfies on average per episode. In IRL fashion, our method solves this problem by adjusting the constraint function iteratively through a constrained optimization procedure, until the agent behavior matches the expert behavior. We demonstrate our approach on synthetic environments, robotics environments and real world highway driving scenarios.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 44 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

17 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022X-MEN: Guaranteed XOR-Maximum Entropy Constrained Inverse Reinforcement Learning22 Mar
  2. 2021Making Human-Like Trade-offs in Constrained Environments by Learning from Demonstrations22 Sep
  3. 2020Deep Inverse Q-learning with Constraints4 Aug
  4. 2019Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning12 Sep
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress18 Jun

Other references · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Density estimation using Real NVPcited by 2,029
  3. 2016OpenAI Gymcited by 1,684
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2018Reward Constrained Policy Optimizationcited by 397
  6. 2018The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systemscited by 276
  7. 2021DC3: A learning method for optimization with hard constraintscited by 181
  8. 2020Inverse Constrained Reinforcement Learningcited by 45
  9. 2019Value constrained model-free continuous controlcited by 34
  10. 2021Maximum Likelihood Constraint Inference from Stochastic Demonstrationscited by 15
  11. 2019Learning Parametric Constraints in High Dimensions from Demonstrationscited by 10
  12. 2020Uncertainty-Aware Constraint Learning for Adaptive Safe Motion Planning from Demonstrationscited by 10

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 6; 2025: 2.

13 papers.

Sort:

By area:

  1. 2023Learning Safety Constraints from Demonstrations with Unknown Rewardscited by 19
  2. 2022Benchmarking Constraint Inference in Inverse Reinforcement Learningcited by 18
  3. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  4. 2023Maximum Causal Entropy Inverse Constrained Reinforcement Learningcited by 7
  5. 2024Confidence Aware Inverse Constrained Reinforcement Learningcited by 4
  6. 2024Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcarecited by 4
  7. 2023Inverse Reinforcement Learning With Constraint Recoverycited by 1
  8. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  9. 2023Learning Safety Constraints From Demonstration Using One-Class Decision Treescited by 0
  10. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraintscited by 0
  11. 2024Learning Soft Driving Constraints from Vectorized Scene Embeddings while Imitating Expert Trajectoriescited by 0
  12. 2025DIAL: Distribution-Informed Adaptive Learning of Multi-Task Constraints for Safety-Critical Systemscited by 0
1 more
  1. 2025DRIVE: Dynamic Rule Inference and Verified Evaluation for Constraint-Aware Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Learning Behavioral Soft Constraints from Demonstrations
  2. 2023Inverse Constraint Learning and Generalization by Transferable Reward Decomposition ↗
  3. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning
  4. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids
  5. 2025Inverse Reinforcement Learning via Convex Optimization
  6. 2024Inverse Reinforcement Learning with Sub-optimal Experts
  7. 2020Inverse Reinforcement Learning from a Gradient-based Learner
  8. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective