# Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/1909.05477 · HTML: https://mlsanity.com/paper/1909.05477/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 3 of its 19 references that are in the map.
- First posted on arXiv 12 Sep 2019.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 3 of 3)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Inferring geometric constraints in human demonstrations](https://mlsanity.com/paper/1810.00140/) (2018)
- [Modeling Supervisor Safe Sets for Improving Collaboration in Human-Robot Teams](https://mlsanity.com/paper/1805.03328/) (2018)

## Built on by (most cited first, 30 of 34)

- [Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysis](https://mlsanity.com/paper/2502.00935/) (2025)
- [Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specification](https://mlsanity.com/paper/2303.08006/) (2023)
- [Sustainable Online Reinforcement Learning for Auto-bidding](https://mlsanity.com/paper/2210.07006/) (2022)
- [Learning Safety Constraints from Demonstrations with Unknown Rewards](https://mlsanity.com/paper/2305.16147/) (2023)
- [Benchmarking Constraint Inference in Inverse Reinforcement Learning](https://mlsanity.com/paper/2206.09670/) (2022)
- [Maximum Likelihood Constraint Inference from Stochastic Demonstrations](https://mlsanity.com/paper/2102.12554/) (2021)
- [Learning Shared Safety Constraints from Multi-task Demonstrations](https://mlsanity.com/paper/2309.00711/) (2023)
- [Learning Soft Constraints From Constrained Expert Demonstrations](https://mlsanity.com/paper/2206.01311/) (2022)
- [A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challenges](https://mlsanity.com/paper/2409.07569/) (2024)
- [Making Human-Like Trade-offs in Constrained Environments by Learning from Demonstrations](https://mlsanity.com/paper/2109.11018/) (2021)
- [Maximum Causal Entropy Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2305.02857/) (2023)
- [Confidence Aware Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2406.16782/) (2024)
- [X-MEN: Guaranteed XOR-Maximum Entropy Constrained Inverse Reinforcement Learning](https://mlsanity.com/paper/2203.11842/) (2022)
- [Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare](https://mlsanity.com/paper/2410.07525/) (2024)
- [Negative Human Rights as a Basis for Long-term AI Safety and Regulation](https://mlsanity.com/paper/2208.14788/) (2022)
- [Your Learned Constraint is Secretly a Backward Reachable Tube](https://mlsanity.com/paper/2501.15618/) (2025)
- [How To Not Drive: Learning Driving Constraints from Demonstration](https://mlsanity.com/paper/2110.00645/) (2021)
- [Inferring Lexicographically-Ordered Rewards from Preferences](https://mlsanity.com/paper/2202.10153/) (2022)
- [Learning Behavioral Soft Constraints from Demonstrations](https://mlsanity.com/paper/2202.10407/) (2022)
- [Inverse Reinforcement Learning With Constraint Recovery](https://mlsanity.com/paper/2305.08130/) (2023)
- [Provably Efficient Exploration in Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2409.15963/) (2024)
- [Learning Neural Control Barrier Functions from Expert Demonstrations using Inverse Constraint Learning](https://mlsanity.com/paper/2510.21560/) (2025)
- [Bayesian Inverse Transition Learning for Offline Settings](https://mlsanity.com/paper/2308.05075/) (2023)
- [Learning Safety Constraints From Demonstration Using One-Class Decision Trees](https://mlsanity.com/paper/2312.08837/) (2023)
- [Positive-Unlabeled Constraint Learning for Inferring Nonlinear Continuous Constraints Functions from Expert Demonstrations](https://mlsanity.com/paper/2408.01622/) (2024)
- [Bayesian Inverse Transition Learning: Learning Dynamics From Near-Optimal Trajectories](https://mlsanity.com/paper/2411.05174/) (2024)
- [DIAL: Distribution-Informed Adaptive Learning of Multi-Task Constraints for Safety-Critical Systems](https://mlsanity.com/paper/2501.18086/) (2025)
- [TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels](https://mlsanity.com/paper/2504.12557/) (2025)
- [Demonstration-Guided Observation Attacks on Black-Box Safe Reinforcement Learning Controllers for Robotic Systems](https://mlsanity.com/paper/2602.16543/) (2026)
- [Safe Reinforcement Learning with Preference-based Constraint Inference](https://mlsanity.com/paper/2603.23565/) (2026)

## Abstract (arXiv)

While most approaches to the problem of Inverse Reinforcement Learning (IRL) focus on estimating a reward function that best explains an expert agent's policy or demonstrated behavior on a control task, it is often the case that such behavior is more succinctly represented by a simple reward combined with a set of hard constraints. In this setting, the agent is attempting to maximize cumulative rewards subject to these given constraints on their behavior. We reformulate the problem of IRL on Markov Decision Processes (MDPs) such that, given a nominal model of the environment and a nominal reward function, we seek to estimate state, action, and feature constraints in the environment that motivate an agent's behavior. Our approach is based on the Maximum Entropy IRL framework, which allows us to reason about the likelihood of an expert agent's demonstrations given our knowledge of an MDP. Using our method, we can infer which constraints can be added to the MDP to most increase the likelihood of observing these demonstrations. We present an algorithm which iteratively infers the Maximum Likelihood Constraint to best explain observed behavior, and we evaluate its efficacy using both simulated behavior and recorded data of humans navigating around an obstacle.