mlsanityData to 14 Oct 2026

Inverse Reinforcement Learning with Sub-optimal Experts

arXiv 2401.03857 ↗First posted 8 Jan 2024cs.LG cs.AIcited by 4

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

also in 1 more
Abstract (arXiv)

Inverse Reinforcement Learning (IRL) techniques deal with the problem of deducing a reward function that explains the behavior of an expert agent who is assumed to act optimally in an underlying unknown task. In several problems of interest, however, it is possible to observe the behavior of multiple experts with different degree of optimality (e.g., racing drivers whose skills ranges from amateurs to professionals). For this reason, in this work, we extend the IRL formulation to problems where, in addition to demonstrations from the optimal agent, we can observe the behavior of multiple sub-optimal experts. Given this problem, we first study the theoretical properties of the class of reward functions that are compatible with a given set of experts, i.e., the feasible reward set. Our results show that the presence of multiple sub-optimal experts can significantly shrink the set of compatible rewards. Furthermore, we study the statistical complexity of estimating the feasible reward set with a generative model. To this end, we analyze a uniform sampling algorithm that results in being minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to the one of the optimal agent.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 11 of its 30 references that are in the map. First posted on arXiv 8 Jan 2024.

Builds on

11 of 30 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees4 Oct
  2. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning22 Sep
  3. 2022Active Exploration for Inverse Reinforcement Learning18 Jul
  4. 2019Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance16 Nov
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress18 Jun

Other references · 6

  1. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  2. 2020Planning in Markov Decision Processes with Gap-Dependent Sample Complexitycited by 34
  3. 2017Repeated Inverse Reinforcement Learningcited by 29
  4. 2019AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teacherscited by 28
  5. 2016Towards Resolving Unidentifiability in Inverse Reinforcement Learningcited by 18
  6. 2023Active Policy Improvement from Multiple Black-box Oraclescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Trainingcited by 9
  2. 2025Efficient Imitation under Misspecificationcited by 5
  3. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroidscited by 1
  4. 2026Quantifying Potential Observation Missingness in Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Inverse Reinforcement Learning via Matching of Optimality Profiles
  2. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning
  3. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  4. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms
  5. 2025Reward Compatibility: A Framework for Inverse RL
  6. 2025Inverse Reinforcement Learning via Convex Optimization
  7. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstrators
  8. 2020Regularized Inverse Reinforcement Learning