# Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms

arXiv: https://arxiv.org/abs/2402.15392 · HTML: https://mlsanity.com/paper/2402.15392/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 10 of its 37 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 10 of 10)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [Is Pessimism Provably Efficient for Offline RL?](https://mlsanity.com/paper/2012.15085/) (2020)
- [Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning](https://mlsanity.com/paper/2106.04895/) (2021)
- [The Importance of Pessimism in Fixed-Dataset Policy Optimization](https://mlsanity.com/paper/2009.06799/) (2020)
- [Active Exploration for Inverse Reinforcement Learning](https://mlsanity.com/paper/2207.08645/) (2022)
- [Towards Theoretical Understanding of Inverse Reinforcement Learning](https://mlsanity.com/paper/2304.12966/) (2023)
- [A Lower Bound for the Sample Complexity of Inverse Reinforcement Learning](https://mlsanity.com/paper/2103.04446/) (2021)
- [Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective](https://mlsanity.com/paper/2312.00054/) (2023)

## Built on by (most cited first, 9 of 9)

- [How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach](https://mlsanity.com/paper/2406.03812/) (2024)
- [On Feasible Rewards in Multi-Agent Inverse Reinforcement Learning](https://mlsanity.com/paper/2411.15046/) (2024)
- [OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatment](https://mlsanity.com/paper/2409.13299/) (2024)
- [Provably Efficient Exploration in Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2409.15963/) (2024)
- [Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids](https://mlsanity.com/paper/2509.12010/) (2025)
- [Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignment](https://mlsanity.com/paper/2602.14844/) (2026)
- [Randomized algorithms and PAC bounds for inverse reinforcement learning in continuous spaces](https://mlsanity.com/paper/2405.15509/) (2024)
- [Robustness in the Face of Partial Identifiability in Reward Learning](https://mlsanity.com/paper/2501.06376/) (2025)
- [Scalable Multi-Task Inverse Reinforcement Learning](https://mlsanity.com/paper/2610.00758/) (2026)

## Abstract (arXiv)

Inverse reinforcement learning (IRL) aims to recover the reward function of an expert agent from demonstrations of behavior. It is well-known that the IRL problem is fundamentally ill-posed, i.e., many reward functions can explain the demonstrations. For this reason, IRL has been recently reframed in terms of estimating the feasible reward set (Metelli et al., 2021), thus, postponing the selection of a single reward. However, so far, the available formulations and algorithmic solutions have been proposed and analyzed mainly for the online setting, where the learner can interact with the environment and query the expert at will. This is clearly unrealistic in most practical applications, where the availability of an offline dataset is a much more common scenario. In this paper, we introduce a novel notion of feasible reward set capturing the opportunities and limitations of the offline setting and we analyze the complexity of its estimation. This requires the introduction an original learning framework that copes with the intrinsic difficulty of the setting, for which the data coverage is not under control. Then, we propose two computationally and statistically efficient algorithms, IRLO and PIRLO, for addressing the problem. In particular, the latter adopts a specific form of pessimism to enforce the novel desirable property of inclusion monotonicity of the delivered feasible set. With this work, we aim to provide a panorama of the challenges of the offline IRL problem and how they can be fruitfully addressed.