mlsanityData to 7 Oct 2026

Inverse Reinforcement Learning by Estimating Expertise of Demonstrators

arXiv 2402.01886 ↗First posted 2 Feb 2024cs.LG cs.AIcited by 11

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

In Imitation Learning (IL), utilizing suboptimal and heterogeneous demonstrations presents a substantial challenge due to the varied nature of real-world data. However, standard IL algorithms consider these datasets as homogeneous, thereby inheriting the deficiencies of suboptimal demonstrators. Previous approaches to this issue rely on impractical assumptions like high-quality data subsets, confidence rankings, or explicit environmental knowledge. This paper introduces IRLEED, Inverse Reinforcement Learning by Estimating Expertise of Demonstrators, a novel framework that overcomes these hurdles without prior knowledge of demonstrator expertise. IRLEED enhances existing Inverse Reinforcement Learning (IRL) algorithms by combining a general model for demonstrator suboptimality to address reward bias and action variance, with a Maximum Entropy IRL framework to efficiently derive the optimal policy from diverse, suboptimal demonstrations. Experiments in both online and offline IL settings, with simulated and human-generated data, demonstrate IRLEED's adaptability and effectiveness, making it a versatile solution for learning from suboptimal demonstrations.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 38 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

22 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models22 Apr

Other references · 21

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  4. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2017Inverse Reward Designcited by 246
  7. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  8. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  9. 2019Imitation Learning from Imperfect Demonstrationcited by 112
  10. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  11. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  12. 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
9 more
  1. 2023Data Quality in Imitation Learningcited by 72
  2. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  3. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
  4. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  5. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamicscited by 26
  6. 2020Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillationcited by 19
  7. 2017The Atari Grand Challenge Datasetcited by 14
  8. 2022Fast Lifelong Adaptive Inverse Reinforcement Learning from Demonstrationscited by 14
  9. 2023Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
  2. 2025Demonstration Sidetracks: Categorizing Systematic Non-Optimality in Human Demonstrationscited by 2
  3. 2025PersonaAgent with GraphRAG: Community-Aware Knowledge Graphs for Personalized LLMcited by 2
  4. 2025LacaDM: A Latent Causal Diffusion Model for Multiobjective Reinforcement Learningcited by 1
  5. 2026REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotationscited by 1
  6. 2026Improving Zero-Shot Offline RL via Behavioral Task Samplingcited by 1
  7. 2025Reinforcement Learning from Multi-level and Episodic Human Feedbackcited by 0
  8. 2025AdversariaL attacK sAfety aLIgnment(ALKALI): Safeguarding LLMs through GRACE: Geometric Representation-Aware Contrastive Enhancement- Introducing Adversarial Vulnerability Quality Index (AVQI)cited by 0
  9. 2026Improving Radio Interferometry Imaging by Explicitly Modeling Cross-Domain Consistency in Reconstructioncited by 0
  10. 2026Robots That Know What to Ask: Recovering Misaligned Rewards through Targeted Explanationscited by 0
  11. 2026Improving Online Reinforcement Learning via Bidirectional Behavior Prior Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Inverse Reinforcement Learning without Reinforcement Learning
  2. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  3. 2024Inverse Reinforcement Learning with Sub-optimal Experts
  4. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  5. 2024Hybrid Inverse Reinforcement Learning
  6. 2019A Divergence Minimization Perspective on Imitation Learning Methods
  7. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment
  8. 2024The Virtues of Pessimism in Inverse Reinforcement Learning