mlsanityData to 14 Oct 2026

Active Exploration for Inverse Reinforcement Learning

arXiv 2207.08645 ↗First posted 18 Jul 2022cs.LG cs.AI stat.MLcited by 27

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse Reinforcement Learning (IRL) is a powerful paradigm for inferring a reward function from expert demonstrations. Many IRL algorithms require a known transition model and sometimes even a known expert policy, or they at least require access to a generative model. However, these assumptions are too strong for many real-world applications, where the environment can be accessed only through sequential interaction. We propose a novel IRL algorithm: Active exploration for Inverse Reinforcement Learning (AceIRL), which actively explores an unknown environment and expert policy to quickly learn the expert's reward function and identify a good policy. AceIRL uses previous observations to construct confidence intervals that capture plausible reward functions and find exploration policies that focus on the most informative regions of the environment. AceIRL is the first approach to active IRL with sample-complexity bounds that does not require a generative model of the environment. AceIRL matches the sample complexity of active IRL with a generative model in the worst case. Additionally, we establish a problem-dependent bound that relates the sample complexity of AceIRL to the suboptimality gap of a given IRL problem. We empirically evaluate AceIRL in simulations and find that it significantly outperforms more naive exploration strategies.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 12 of its 38 references that are in the map. First posted on arXiv 18 Jul 2022.

Builds on

12 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Inverse Reinforcement Learning in a Continuous State Space with Formal Guarantees16 Feb
  2. 2019Risk-Aware Active Inverse Reinforcement Learning8 Jan

Other references · 10

  1. 2016Concrete Problems in AI Safetycited by 1,642
  2. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  3. 2021Unsolved Problems in ML Safetycited by 292
  4. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  5. 2018Learning to Walk via Deep Reinforcement Learningcited by 158
  6. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  7. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  8. 2020Adaptive Reward-Free Explorationcited by 77
  9. 2021Online Apprenticeship Learningcited by 29
  10. 2018Including Uncertainty when Learning from Human Correctionscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 12; 2025: 6; 2026: 1.

27 papers.

Sort:

By area:

  1. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 148
  2. 2023Towards Theoretical Understanding of Inverse Reinforcement Learningcited by 21
  3. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitionscited by 16
  4. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  5. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithmscited by 9
  6. 2023Reinforcement Learning-assisted Evolutionary Algorithm: A Survey and Research Opportunitiescited by 8
  7. 2024Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstractioncited by 6
  8. 2023Curricular Subgoals for Inverse Reinforcement Learningcited by 5
  9. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approachcited by 5
  10. 2022Environment Design for Inverse Reinforcement Learningcited by 4
  11. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspectivecited by 4
  12. 2024Inverse Reinforcement Learning with Sub-optimal Expertscited by 4
15 more
  1. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedbackcited by 4
  2. 2024On Feasible Rewards in Multi-Agent Inverse Reinforcement Learningcited by 4
  3. 2024Preference Elicitation for Offline Reinforcement Learningcited by 3
  4. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignmentcited by 3
  5. 2024Inverse Concave-Utility Reinforcement Learning is Inverse Game Theorycited by 2
  6. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  7. 2025Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Reviewcited by 1
  8. 2025Apprenticeship learning with prior beliefs using inverse optimizationcited by 1
  9. 2025PAC Apprenticeship Learning with Bayesian Active Inverse Reinforcement Learningcited by 1
  10. 2025Efficient Reward Identification In Max Entropy Reinforcement Learning with Sparsity and Rank Priorscited by 1
  11. 2023PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Rewardcited by 0
  12. 2024Randomized algorithms and PAC bounds for inverse reinforcement learning in continuous spacescited by 0
  13. 2025Robustness in the Face of Partial Identifiability in Reward Learningcited by 0
  14. 2025Efficient Personalization of Generative Models via Optimal Experimental Designcited by 0
  15. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approachcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching
  2. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  3. 2024The Virtues of Pessimism in Inverse Reinforcement Learning
  4. 2026Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimization
  5. 2023Inverse Reinforcement Learning without Reinforcement Learning
  6. 2024Accelerating Inverse Reinforcement Learning with Expert Bootstrapping
  7. 2025Inverse Reinforcement Learning via Convex Optimization
  8. 2024Toward Information Theoretic Active Inverse Reinforcement Learning