# Benchmarking Constraint Inference in Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2206.09670 · HTML: https://mlsanity.com/paper/2206.09670/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 20 Jun 2022.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems](https://mlsanity.com/paper/1810.05642/) (2018)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Projection-Based Constrained Policy Optimization](https://mlsanity.com/paper/2010.03152/) (2020)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Imitation Learning via Off-Policy Distribution Matching](https://mlsanity.com/paper/1912.05032/) (2019)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [IPO: Interior-point Policy Optimization under Constraints](https://mlsanity.com/paper/1910.09615/) (2019)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Learning from Suboptimal Demonstration via Self-Supervised Reward Regression](https://mlsanity.com/paper/2010.11723/) (2020)
- [Scalable Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2102.06483/) (2021)
- [Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2011.09999/) (2020)
- [Dirichlet Variational Autoencoder](https://mlsanity.com/paper/1901.02739/) (2019)
- [Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning](https://mlsanity.com/paper/1909.05477/) (2019)
- [Truly Batch Apprenticeship Learning with Deep Successor Features](https://mlsanity.com/paper/1903.10077/) (2019)
- [Robust Imitation Learning from Noisy Demonstrations](https://mlsanity.com/paper/2010.10181/) (2020)
- [Maximum Likelihood Constraint Inference from Stochastic Demonstrations](https://mlsanity.com/paper/2102.12554/) (2021)
- [Learning Soft Constraints From Constrained Expert Demonstrations](https://mlsanity.com/paper/2206.01311/) (2022)
- [Learning Parametric Constraints in High Dimensions from Demonstrations](https://mlsanity.com/paper/1910.03477/) (2019)
- [Uncertainty-Aware Constraint Learning for Adaptive Safe Motion Planning from Demonstrations](https://mlsanity.com/paper/2011.04141/) (2020)
- [Learning Constraints from Demonstrations](https://mlsanity.com/paper/1812.07084/) (2018)
- [Making Human-Like Trade-offs in Constrained Environments by Learning from Demonstrations](https://mlsanity.com/paper/2109.11018/) (2021)

## Built on by (most cited first, 18 of 18)

- [Learning Safety Constraints from Demonstrations with Unknown Rewards](https://mlsanity.com/paper/2305.16147/) (2023)
- [An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient](https://mlsanity.com/paper/2307.08873/) (2023)
- [A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challenges](https://mlsanity.com/paper/2409.07569/) (2024)
- [Maximum Causal Entropy Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2305.02857/) (2023)
- [A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization](https://mlsanity.com/paper/2403.11062/) (2024)
- [Confidence Aware Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2406.16782/) (2024)
- [Safety through feedback in Constrained RL](https://mlsanity.com/paper/2406.19626/) (2024)
- [Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare](https://mlsanity.com/paper/2410.07525/) (2024)
- [Provably Efficient Exploration in Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2409.15963/) (2024)
- [Learning Safety Constraints From Demonstration Using One-Class Decision Trees](https://mlsanity.com/paper/2312.08837/) (2023)
- [Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints](https://mlsanity.com/paper/2405.03005/) (2024)
- [Positive-Unlabeled Constraint Learning for Inferring Nonlinear Continuous Constraints Functions from Expert Demonstrations](https://mlsanity.com/paper/2408.01622/) (2024)
- [Learning Soft Driving Constraints from Vectorized Scene Embeddings while Imitating Expert Trajectories](https://mlsanity.com/paper/2412.05717/) (2024)
- [DIAL: Distribution-Informed Adaptive Learning of Multi-Task Constraints for Safety-Critical Systems](https://mlsanity.com/paper/2501.18086/) (2025)
- [Measures of Variability for Risk-averse Policy Gradient](https://mlsanity.com/paper/2504.11412/) (2025)
- [TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels](https://mlsanity.com/paper/2504.12557/) (2025)
- [DRIVE: Dynamic Rule Inference and Verified Evaluation for Constraint-Aware Autonomous Driving](https://mlsanity.com/paper/2508.04066/) (2025)
- [Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination](https://mlsanity.com/paper/2610.02170/) (2026)

## Abstract (arXiv)

When deploying Reinforcement Learning (RL) agents into a physical system, we must ensure that these agents are well aware of the underlying constraints. In many real-world problems, however, the constraints are often hard to specify mathematically and unknown to the RL agents. To tackle these issues, Inverse Constrained Reinforcement Learning (ICRL) empirically estimates constraints from expert demonstrations. As an emerging research topic, ICRL does not have common benchmarks, and previous works tested algorithms under hand-crafted environments with manually-generated expert demonstrations. In this paper, we construct an ICRL benchmark in the context of RL application domains, including robot control, and autonomous driving. For each environment, we design relevant constraints and train expert agents to generate demonstration data. Besides, unlike existing baselines that learn a deterministic constraint, we propose a variational ICRL method to model a posterior distribution of candidate constraints. We conduct extensive experiments on these algorithms under our benchmark and show how they can facilitate studying important research challenges for ICRL. The benchmark, including the instructions for reproducing ICRL algorithms, is available at https://github.com/Guiliang/ICRL-benchmarks-public.