# Inverse Reinforcement Learning With Constraint Recovery

arXiv: https://arxiv.org/abs/2305.08130 · HTML: https://mlsanity.com/paper/2305.08130/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 14 May 2023.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 9 of 9)

- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Introduction to Online Convex Optimization](https://mlsanity.com/paper/1909.05207/) (2019)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2011.09999/) (2020)
- [Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning](https://mlsanity.com/paper/1909.05477/) (2019)
- [Deep Inverse Q-learning with Constraints](https://mlsanity.com/paper/2008.01712/) (2020)
- [Learning Soft Constraints From Constrained Expert Demonstrations](https://mlsanity.com/paper/2206.01311/) (2022)
- [Uncertainty-Aware Constraint Learning for Adaptive Safe Motion Planning from Demonstrations](https://mlsanity.com/paper/2011.04141/) (2020)
- [X-MEN: Guaranteed XOR-Maximum Entropy Constrained Inverse Reinforcement Learning](https://mlsanity.com/paper/2203.11842/) (2022)

## Built on by (most cited first, 1 of 1)

- [Generalized Exponentiated Gradient Algorithms and Their Application to On-Line Portfolio Selection](https://mlsanity.com/paper/2406.00655/) (2024)

## Abstract (arXiv)

In this work, we propose a novel inverse reinforcement learning (IRL) algorithm for constrained Markov decision process (CMDP) problems. In standard IRL problems, the inverse learner or agent seeks to recover the reward function of the MDP, given a set of trajectory demonstrations for the optimal policy. In this work, we seek to infer not only the reward functions of the CMDP, but also the constraints. Using the principle of maximum entropy, we show that the IRL with constraint recovery (IRL-CR) problem can be cast as a constrained non-convex optimization problem. We reduce it to an alternating constrained optimization problem whose sub-problems are convex. We use exponentiated gradient descent algorithm to solve it. Finally, we demonstrate the efficacy of our algorithm for the grid world environment.