mlsanityData to 14 Oct 2026

Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach

arXiv 2605.30903 ↗First posted 29 May 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

also in 1 more
Abstract (arXiv)

Inverse reinforcement learning (IRL) typically assumes demonstrations from a single optimal demonstrator, but in many applications data come from multiple imperfect demonstrators with heterogeneous suboptimality levels. We study reward learning in this setting through a feasible-reward-set framework: for each demonstrator, we encode its declared suboptimality level as a linear constraint and intersect the resulting feasible sets across demonstrators. Our theoretical analysis shows that the joint feasible set shrinks monotonically as data are added, and we give an exact characterization of when a new demonstrator strictly tightens it. We further establish two recovery guarantees for the feasible reward set of the ground-truth optimal demonstrator: one bound depends on closeness to the optimal occupancy, while the other requires only sufficient coverage and no near-optimal demonstrator. On the practical side, we introduce strategies to address the inherent reward ambiguity in the obtained reward set and provide an offline algorithm with function approximation for high-dimensional environments. Experiments in tabular grid-world and large language model (LLM) fine-tuning settings are consistent with the theoretical predictions and demonstrate the effectiveness of the proposed framework over baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 52 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

25 of 52 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids15 Sep
  2. 2024On Feasible Rewards in Multi-Agent Inverse Reinforcement Learning22 Nov
  3. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedback20 May
  4. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning25 Apr
  5. 2022Active Exploration for Inverse Reinforcement Learning18 Jul
  6. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations12 Apr

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  8. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  9. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 118
  10. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  11. 2018Reinforcement Learning from Imperfect Demonstrationscited by 75
  12. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 74
7 more
  1. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  2. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  3. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
  4. 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
  5. 2020Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillationcited by 19
  6. 2022A Ranking Game for Imitation Learningcited by 18
  7. 2019New characterizations of Hoffman constants for systems of linear constraintscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Compatibility: A Framework for Inverse RL
  2. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  3. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  4. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach
  5. 2023A Novel Variational Lower Bound for Inverse Reinforcement Learning
  6. 2026A Geometric Perspective on Reward Function Updates in Inverse Reinforcement Learning
  7. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms
  8. 2020Inverse Reinforcement Learning via Matching of Optimality Profiles