mlsanityData to 14 Oct 2026

Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective

arXiv 2312.00054 ↗First posted 29 Nov 2023stat.ML cs.AI cs.LGcited by 4

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse Reinforcement Learning (IRL) -- the problem of learning reward functions from demonstrations of an expert policy -- plays a critical role in developing intelligent systems. While widely used in applications, theoretical understandings of IRL present unique challenges and remain less developed compared with standard RL. For example, it remains open how to do IRL efficiently in standard offline settings with pre-collected data, where states are obtained from a behavior policy (which could be the expert policy itself), and actions are sampled from the expert policy. This paper provides the first line of results for efficient IRL in vanilla offline and online settings using polynomial samples and runtime. Our algorithms and analyses seamlessly adapt the pessimism principle commonly used in offline RL, and achieve IRL guarantees in stronger metrics than considered in existing work. We provide lower bounds showing that our sample complexities are nearly optimal. As an application, we also show that the learned rewards can transfer to another target MDP with suitable guarantees when the target MDP satisfies certain similarity assumptions with the original (source) MDP.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 76 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

38 of 76 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Massively Scalable Inverse Reinforcement Learning in Google Maps18 May
  2. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning25 Apr
  3. 2022Active Exploration for Inverse Reinforcement Learning18 Jul
  4. 2021Inverse Reinforcement Learning in a Continuous State Space with Formal Guarantees16 Feb
  5. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations12 Apr
  6. 2018Adversarial Imitation via Variational Inverse Reinforcement Learning17 Sep

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2016Generative Adversarial Imitation Learningcited by 1,597
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  9. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  10. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  11. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  12. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
20 more
  1. 2016Cooperative Inverse Reinforcement Learningcited by 300
  2. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  3. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  4. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  5. 2020Almost Optimal Model-Free Reinforcement Learning via Reference-Advantage Decompositioncited by 127
  6. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 118
  7. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  8. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  9. 2023Is RLHF More Difficult than Standard RL?cited by 96
  10. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  11. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  12. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  13. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  14. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  15. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 46
  16. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  17. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  18. 2023Minimax-Optimal Reward-Agnostic Exploration in Reinforcement Learningcited by 15
  19. 2017Agile Autonomous Driving using End-to-End Deep Imitation Learningcited by 10
  20. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preferencecited by 10

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithmscited by 9
  2. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approachcited by 5
  3. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  4. 2025Robustness in the Face of Partial Identifiability in Reward Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Virtues of Pessimism in Inverse Reinforcement Learning
  2. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  3. 2023Inverse Reinforcement Learning without Reinforcement Learning
  4. 2025Reward Compatibility: A Framework for Inverse RL
  5. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
  6. 2024Hybrid Inverse Reinforcement Learning
  7. 2018A Framework and Method for Online Inverse Reinforcement Learning
  8. 2021A Generalised Inverse Reinforcement Learning Framework