mlsanityData to 7 Oct 2026

Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning

arXiv 1909.05477 ↗First posted 12 Sep 2019cs.LG cs.AI cs.ROcited by 33

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

While most approaches to the problem of Inverse Reinforcement Learning (IRL) focus on estimating a reward function that best explains an expert agent's policy or demonstrated behavior on a control task, it is often the case that such behavior is more succinctly represented by a simple reward combined with a set of hard constraints. In this setting, the agent is attempting to maximize cumulative rewards subject to these given constraints on their behavior. We reformulate the problem of IRL on Markov Decision Processes (MDPs) such that, given a nominal model of the environment and a nominal reward function, we seek to estimate state, action, and feature constraints in the environment that motivate an agent's behavior. Our approach is based on the Maximum Entropy IRL framework, which allows us to reason about the likelihood of an expert agent's demonstrations given our knowledge of an MDP. Using our method, we can infer which constraints can be added to the MDP to most increase the likelihood of observing these demonstrations. We present an algorithm which iteratively infers the Maximum Likelihood Constraint to best explain observed behavior, and we evaluate its efficacy using both simulated behavior and recorded data of humans navigating around an obstacle.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 3 of its 19 references that are in the map. First posted on arXiv 12 Sep 2019.

Builds on

3 of 19 references.

References · 3

  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2018Inferring geometric constraints in human demonstrationscited by 5
  3. 2018Modeling Supervisor Safe Sets for Improving Collaboration in Human-Robot Teamscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 7; 2024: 6; 2025: 5; 2026: 5.

33 papers.

Sort:

By area:

  1. 2025Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysiscited by 44
  2. 2023Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specificationcited by 38
  3. 2022Sustainable Online Reinforcement Learning for Auto-biddingcited by 28
  4. 2023Learning Safety Constraints from Demonstrations with Unknown Rewardscited by 19
  5. 2022Benchmarking Constraint Inference in Inverse Reinforcement Learningcited by 18
  6. 2021Maximum Likelihood Constraint Inference from Stochastic Demonstrationscited by 15
  7. 2023Learning Shared Safety Constraints from Multi-task Demonstrationscited by 15
  8. 2022Learning Soft Constraints From Constrained Expert Demonstrationscited by 13
  9. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  10. 2021Making Human-Like Trade-offs in Constrained Environments by Learning from Demonstrationscited by 7
  11. 2023Maximum Causal Entropy Inverse Constrained Reinforcement Learningcited by 7
  12. 2022X-MEN: Guaranteed XOR-Maximum Entropy Constrained Inverse Reinforcement Learningcited by 4
21 more
  1. 2024Confidence Aware Inverse Constrained Reinforcement Learningcited by 4
  2. 2024Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcarecited by 4
  3. 2022Negative Human Rights as a Basis for Long-term AI Safety and Regulationcited by 3
  4. 2021How To Not Drive: Learning Driving Constraints from Demonstrationcited by 2
  5. 2022Inferring Lexicographically-Ordered Rewards from Preferencescited by 2
  6. 2022Learning Behavioral Soft Constraints from Demonstrationscited by 2
  7. 2025Your Learned Constraint is Secretly a Backward Reachable Tubecited by 2
  8. 2023Inverse Reinforcement Learning With Constraint Recoverycited by 1
  9. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  10. 2023Bayesian Inverse Transition Learning for Offline Settingscited by 0
  11. 2023Learning Safety Constraints From Demonstration Using One-Class Decision Treescited by 0
  12. 2024Positive-Unlabeled Constraint Learning for Inferring Nonlinear Continuous Constraints Functions from Expert Demonstrationscited by 0
  13. 2024Bayesian Inverse Transition Learning: Learning Dynamics From Near-Optimal Trajectoriescited by 0
  14. 2025DIAL: Distribution-Informed Adaptive Learning of Multi-Task Constraints for Safety-Critical Systemscited by 0
  15. 2025TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labelscited by 0
  16. 2025Learning Neural Control Barrier Functions from Expert Demonstrations using Inverse Constraint Learningcited by 0
  17. 2026Demonstration-Guided Observation Attacks on Black-Box Safe Reinforcement Learning Controllers for Robotic Systemscited by 0
  18. 2026Safe Reinforcement Learning with Preference-based Constraint Inferencecited by 0
  19. 2026Multi-Objective Constraint Inference using Inverse reinforcement learningcited by 0
  20. 2026Failing Gracefully: Mitigating Impact of Inevitable Robot Failurescited by 0
  21. 2026Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordinationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  2. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees
  3. 2021Task-Guided Inverse Reinforcement Learning Under Partial Information
  4. 2023FP-IRL: Fokker--Planck Inverse Reinforcement Learning -- A Physics-Constrained Approach to Markov Decision Processes
  5. 2022Task-Guided IRL in POMDPs that Scales
  6. 2024Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards
  7. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  8. 2019Generalized Maximum Causal Entropy for Inverse Reinforcement Learning