# Towards Generalized Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2402.07246 · HTML: https://mlsanity.com/paper/2402.07246/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 11 Feb 2024.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 7 of 7)

- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Model-Free Imitation Learning with Policy Optimization](https://mlsanity.com/paper/1605.08478/) (2016)
- [Repeated Inverse Reinforcement Learning](https://mlsanity.com/paper/1705.05427/) (2017)
- [Generalized Inverse Optimization through Online Learning](https://mlsanity.com/paper/1810.01920/) (2018)
- [Wasserstein Distributionally Robust Inverse Multiobjective Optimization](https://mlsanity.com/paper/2009.14552/) (2020)
- [Inferring Parameters Through Inverse Multiobjective Optimization](https://mlsanity.com/paper/1808.00935/) (2018)

## Built on by (most cited first, 2 of 2)

- [Inverse Reinforcement Learning for Minimum-Exposure Paths in Spatiotemporally Varying Scalar Fields](https://mlsanity.com/paper/2503.06611/) (2025)
- [Inference of Utilities and Time Preference in Sequential Decision-Making](https://mlsanity.com/paper/2405.15975/) (2024)

## Abstract (arXiv)

This paper studies generalized inverse reinforcement learning (GIRL) in Markov decision processes (MDPs), that is, the problem of learning the basic components of an MDP given observed behavior (policy) that might not be optimal. These components include not only the reward function and transition probability matrices, but also the action space and state space that are not exactly known but are known to belong to given uncertainty sets. We address two key challenges in GIRL: first, the need to quantify the discrepancy between the observed policy and the underlying optimal policy; second, the difficulty of mathematically characterizing the underlying optimal policy when the basic components of an MDP are unobservable or partially observable. Then, we propose the mathematical formulation for GIRL and develop a fast heuristic algorithm. Numerical results on both finite and infinite state problems show the merit of our formulation and algorithm.