# X-MEN: Guaranteed XOR-Maximum Entropy Constrained Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2203.11842 · HTML: https://mlsanity.com/paper/2203.11842/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 16 of 16)

- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2011.09999/) (2020)
- [Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning](https://mlsanity.com/paper/1909.05477/) (2019)
- [Deep Inverse Q-learning with Constraints](https://mlsanity.com/paper/2008.01712/) (2020)
- [Distribution-Aware Sampling and Weighted Model Counting for SAT](https://mlsanity.com/paper/1404.2984/) (2014)
- [Maximum Likelihood Constraint Inference from Stochastic Demonstrations](https://mlsanity.com/paper/2102.12554/) (2021)
- [Taming the Curse of Dimensionality: Discrete Integration by Hashing and Optimization](https://mlsanity.com/paper/1302.6677/) (2013)
- [Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms](https://mlsanity.com/paper/2012.00889/) (2020)
- [Learning Constraints from Demonstrations](https://mlsanity.com/paper/1812.07084/) (2018)
- [Inferring geometric constraints in human demonstrations](https://mlsanity.com/paper/1810.00140/) (2018)
- [Modeling Supervisor Safe Sets for Improving Collaboration in Human-Robot Teams](https://mlsanity.com/paper/1805.03328/) (2018)
- [Towards Efficient Discrete Integration via Adaptive Quantile Queries](https://mlsanity.com/paper/1910.05811/) (2019)

## Built on by (most cited first, 4 of 4)

- [Identifiability and Generalizability in Constrained Inverse Reinforcement Learning](https://mlsanity.com/paper/2306.00629/) (2023)
- [Learning Soft Constraints From Constrained Expert Demonstrations](https://mlsanity.com/paper/2206.01311/) (2022)
- [A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challenges](https://mlsanity.com/paper/2409.07569/) (2024)
- [Inverse Reinforcement Learning With Constraint Recovery](https://mlsanity.com/paper/2305.08130/) (2023)

## Abstract (arXiv)

Inverse Reinforcement Learning (IRL) is a powerful way of learning from demonstrations. In this paper, we address IRL problems with the availability of prior knowledge that optimal policies will never violate certain constraints. Conventional approaches ignoring these constraints need many demonstrations to converge. We propose XOR-Maximum Entropy Constrained Inverse Reinforcement Learning (X-MEN), which is guaranteed to converge to the optimal policy in linear rate w.r.t. the number of learning iterations. X-MEN embeds XOR-sampling -- a provable sampling approach that transforms the #P complete sampling problem into queries to NP oracles -- into the framework of maximum entropy IRL. X-MEN also guarantees the learned policy will never generate trajectories that violate constraints. Empirical results in navigation demonstrate that X-MEN converges faster to the optimal policies compared to baseline approaches and always generates trajectories that satisfy multi-state combinatorial constraints.