mlsanityData to 7 Oct 2026

Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms

arXiv 2402.15392 ↗First posted 23 Feb 2024cs.LGcited by 9

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse reinforcement learning (IRL) aims to recover the reward function of an expert agent from demonstrations of behavior. It is well-known that the IRL problem is fundamentally ill-posed, i.e., many reward functions can explain the demonstrations. For this reason, IRL has been recently reframed in terms of estimating the feasible reward set (Metelli et al., 2021), thus, postponing the selection of a single reward. However, so far, the available formulations and algorithmic solutions have been proposed and analyzed mainly for the online setting, where the learner can interact with the environment and query the expert at will. This is clearly unrealistic in most practical applications, where the availability of an offline dataset is a much more common scenario. In this paper, we introduce a novel notion of feasible reward set capturing the opportunities and limitations of the offline setting and we analyze the complexity of its estimation. This requires the introduction an original learning framework that copes with the intrinsic difficulty of the setting, for which the data coverage is not under control. Then, we propose two computationally and statistically efficient algorithms, IRLO and PIRLO, for addressing the problem. In particular, the latter adopts a specific form of pessimism to enforce the novel desirable property of inclusion monotonicity of the delivered feasible set. With this work, we aim to provide a panorama of the challenges of the offline IRL problem and how they can be fruitfully addressed.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 10 of its 37 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

10 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective29 Nov
  2. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning25 Apr
  3. 2022Active Exploration for Inverse Reinforcement Learning18 Jul
  4. 2021A Lower Bound for the Sample Complexity of Inverse Reinforcement Learning7 Mar

Other references · 6

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  4. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  5. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  6. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 2.

9 papers.

  1. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approachcited by 5
  2. 2024On Feasible Rewards in Multi-Agent Inverse Reinforcement Learningcited by 4
  3. 2024OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatmentcited by 1
  4. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  5. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroidscited by 1
  6. 2026Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignmentcited by 1
  7. 2024Randomized algorithms and PAC bounds for inverse reinforcement learning in continuous spacescited by 0
  8. 2025Robustness in the Face of Partial Identifiability in Reward Learningcited by 0
  9. 2026Scalable Multi-Task Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  2. 2024The Virtues of Pessimism in Inverse Reinforcement Learning
  3. 2023Inverse Reinforcement Learning without Reinforcement Learning
  4. 2024Inverse Reinforcement Learning with Sub-optimal Experts
  5. 2018A Framework and Method for Online Inverse Reinforcement Learning
  6. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
  7. 2021Offline Inverse Reinforcement Learning
  8. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning