mlsanityData to 7 Oct 2026

Towards Theoretical Understanding of Inverse Reinforcement Learning

arXiv 2304.12966 ↗First posted 25 Apr 2023cs.LGcited by 21

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse reinforcement learning (IRL) denotes a powerful family of algorithms for recovering a reward function justifying the behavior demonstrated by an expert agent. A well-known limitation of IRL is the ambiguity in the choice of the reward function, due to the existence of multiple rewards that explain the observed behavior. This limitation has been recently circumvented by formulating IRL as the problem of estimating the feasible reward set, i.e., the region of the rewards compatible with the expert's behavior. In this paper, we make a step towards closing the theory gap of IRL in the case of finite-horizon problems with a generative model. We start by formally introducing the problem of estimating the feasible reward set, the corresponding PAC requirement, and discussing the properties of particular classes of rewards. Then, we provide the first minimax lower bound on the sample complexity for the problem of estimating the feasible reward set of order OmegaBigl( fracH3SAε2 bigl( log bigl(frac1δbigl) + S bigl)Bigl), being S and A the number of states and actions respectively, H the horizon, ε the desired accuracy, and δ the confidence. We analyze the sample complexity of a uniform sampling strategy (US-IRL), proving a matching upper bound up to logarithmic factors. Finally, we outline several open questions in IRL and propose future research directions.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 16 of its 44 references that are in the map. First posted on arXiv 25 Apr 2023.

Builds on

16 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees4 Oct
  2. 2022Active Exploration for Inverse Reinforcement Learning18 Jul
  3. 2021A Lower Bound for the Sample Complexity of Inverse Reinforcement Learning7 Mar
  4. 2021Inverse Reinforcement Learning in a Continuous State Space with Formal Guarantees16 Feb
  5. 2019On the Correctness and Sample Complexity of Inverse Reinforcement Learning2 Jun
  6. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress18 Jun

Other references · 10

  1. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  2. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
  3. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  4. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
  5. 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 104
  6. 2020Fast active learning for pure exploration in reinforcement learningcited by 92
  7. 2020Adaptive Reward-Free Explorationcited by 77
  8. 2020Planning in Markov Decision Processes with Gap-Dependent Sample Complexitycited by 34
  9. 2020Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning with Application to Autonomous Drivingcited by 34
  10. 2016Learning to Drive using Inverse Reinforcement Learning and Deep Q-Networkscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 6; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinementcited by 11
  2. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  3. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithmscited by 9
  4. 2023Inverse Reinforcement Learning with the Average Reward Criterioncited by 8
  5. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approachcited by 5
  6. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspectivecited by 4
  7. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedbackcited by 4
  8. 2024On Feasible Rewards in Multi-Agent Inverse Reinforcement Learningcited by 4
  9. 2024Inverse Concave-Utility Reinforcement Learning is Inverse Game Theorycited by 2
  10. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  11. 2025PAC Apprenticeship Learning with Bayesian Active Inverse Reinforcement Learningcited by 1
  12. 2025Efficient Reward Identification In Max Entropy Reinforcement Learning with Sparsity and Rank Priorscited by 1
9 more
  1. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroidscited by 1
  2. 2024Randomized algorithms and PAC bounds for inverse reinforcement learning in continuous spacescited by 0
  3. 2025Robustness in the Face of Partial Identifiability in Reward Learningcited by 0
  4. 2025Apprenticeship learning with prior beliefs using inverse optimizationcited by 0
  5. 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learningcited by 0
  6. 2026Learning Incentive Structures for Cooperative Resilience in Multi-Agent Systems under Social Dilemmascited by 0
  7. 2026Learning the Preferences of a Learning Agentcited by 0
  8. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approachcited by 0
  9. 2026Path-dependent Discrete Amortized Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Compatibility: A Framework for Inverse RL
  2. 2024Inverse Reinforcement Learning with Sub-optimal Experts
  3. 2025Inverse Reinforcement Learning via Convex Optimization
  4. 2023A Novel Variational Lower Bound for Inverse Reinforcement Learning
  5. 2024Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards
  6. 2019Option Compatible Reward Inverse Reinforcement Learning
  7. 2020Inverse Reinforcement Learning via Matching of Optimality Profiles
  8. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization