# Learning Behavioral Soft Constraints from Demonstrations

arXiv: https://arxiv.org/abs/2202.10407 · HTML: https://mlsanity.com/paper/2202.10407/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 29 references that are in the map.
- First posted on arXiv 21 Feb 2022.

## Builds on (most cited first, 7 of 7)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Research Priorities for Robust and Beneficial Artificial Intelligence](https://mlsanity.com/paper/1602.03506/) (2016)
- [Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2011.09999/) (2020)
- [Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning](https://mlsanity.com/paper/1909.05477/) (2019)
- [Building Ethically Bounded AI](https://mlsanity.com/paper/1812.03980/) (2018)
- [A Low-Cost Ethics Shaping Approach for Designing Reinforcement Learning Agents](https://mlsanity.com/paper/1712.04172/) (2017)
- [Incorporating Behavioral Constraints in Online AI Systems](https://mlsanity.com/paper/1809.05720/) (2018)

## Built on by (most cited first, 2 of 2)

- [Maximum Causal Entropy Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2305.02857/) (2023)
- [Positive-Unlabeled Constraint Learning for Inferring Nonlinear Continuous Constraints Functions from Expert Demonstrations](https://mlsanity.com/paper/2408.01622/) (2024)

## Abstract (arXiv)

Many real-life scenarios require humans to make difficult trade-offs: do we always follow all the traffic rules or do we violate the speed limit in an emergency? These scenarios force us to evaluate the trade-off between collective rules and norms with our own personal objectives and desires. To create effective AI-human teams, we must equip AI agents with a model of how humans make these trade-offs in complex environments when there are implicit and explicit rules and constraints. Agent equipped with these models will be able to mirror human behavior and/or to draw human attention to situations where decision making could be improved. To this end, we propose a novel inverse reinforcement learning (IRL) method: Max Entropy Inverse Soft Constraint IRL (MESC-IRL), for learning implicit hard and soft constraints over states, actions, and state features from demonstrations in deterministic and non-deterministic environments modeled as Markov Decision Processes (MDPs). Our method enables agents implicitly learn human constraints and desires without the need for explicit modeling by the agent designer and to transfer these constraints between environments. Our novel method generalizes prior work which only considered deterministic hard constraints and achieves state of the art performance.