mlsanityData to 7 Oct 2026

Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations

arXiv 1904.06387 ↗First posted 12 Apr 2019cs.LG stat.MLcited by 256

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

A critical flaw of existing inverse reinforcement learning (IRL) methods is their inability to significantly outperform the demonstrator. This is because IRL typically seeks a reward function that makes the demonstrator appear near-optimal, rather than inferring the underlying intentions of the demonstrator that may have been poorly executed in practice. In this paper, we introduce a novel reward-learning-from-observation algorithm, Trajectory-ranked Reward EXtrapolation (T-REX), that extrapolates beyond a set of (approximately) ranked demonstrations in order to infer high-quality reward functions from a set of potentially poor demonstrations. When combined with deep reinforcement learning, T-REX outperforms state-of-the-art imitation learning and IRL methods on multiple Atari and MuJoCo benchmark tasks and achieves performance that is often more than twice the performance of the best demonstration. We also demonstrate that T-REX is robust to ranking noise and can accurately extrapolate intention by simply watching a learner noisily improve at a task over time.

Key facts. Cited by 256 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 61 references that are in the map. First posted on arXiv 12 Apr 2019.

Builds on

25 of 61 references.

Closest ancestors · 3

Same direction, newest first

  1. 2018Inverse reinforcement learning for video games24 Oct
  2. 2018Adversarial Imitation via Variational Inverse Reinforcement Learning17 Sep
  3. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress18 Jun

Other references · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2016OpenAI Gymcited by 1,684
  5. 2016Concrete Problems in AI Safetycited by 1,582
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2018Behavioral Cloning from Observationcited by 482
  9. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  10. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  11. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
10 more
  1. 2018Generative Adversarial Imitation from Observationcited by 183
  2. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  3. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  4. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  5. 2017Visualizing and Understanding Atari Agentscited by 141
  6. 2018Playing hard exploration games by watching YouTubecited by 98
  7. 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
  8. 2017OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learningcited by 40
  9. 2018One-Shot Learning of Multi-Step Tasks from Observation via Activity Localization in Auxiliary Videocited by 16
  10. 2017The Atari Grand Challenge Datasetcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 5; 2020: 10; 2021: 29; 2022: 43; 2023: 56; 2024: 47; 2025: 41; 2026: 25.

256 papers.

Sort:

By area:

  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  4. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  5. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  7. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  8. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
  9. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  10. 2023Vision-Language Models as Success Detectorscited by 138
  11. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  12. 2020Primal Wasserstein Imitation Learningcited by 120
38 more
  1. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  2. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  3. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  4. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  5. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  6. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  7. 2021Off-Policy Imitation Learning from Observationscited by 70
  8. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  9. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  10. 2022Adversarial Training for High-Stakes Reliabilitycited by 60
  11. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  12. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 56
  13. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  14. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  15. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  16. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 52
  17. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 50
  18. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  19. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
  20. 2021Learning Multimodal Rewards from Rankingscited by 41
  21. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  22. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 38
  23. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
  24. 2020Intrinsic Reward Driven Imitation Learning via Generative Modelcited by 35
  25. 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 34
  26. 2024Aligning Language Models with Demonstrated Feedbackcited by 32
  27. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  28. 2023Second Thoughts are Best: Learning to Re-Align With Human Values from Text Editscited by 27
  29. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  30. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  31. 2020Robust Imitation Learning from Noisy Demonstrationscited by 26
  32. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
  33. 2021Learning Reward Functions from Scale Feedbackcited by 24
  34. 2021MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learningcited by 24
  35. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  36. 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningcited by 23
  37. 2023A density estimation perspective on learning from pairwise human preferencescited by 23
  38. 2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learningcited by 22

50 most cited of 256 All 256 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2019Deep Bayesian Reward Learning from Preferences
  2. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
  3. 2025Beyond Imitation: Recovering Dense Rewards from Demonstrations
  4. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
  5. 2022IL-flOw: Imitation Learning from Observation using Normalizing Flows
  6. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variables
  7. 2025TW-CRL: Time-Weighted Contrastive Reward Learning for Efficient Inverse Reinforcement Learning
  8. 2019Option Compatible Reward Inverse Reinforcement Learning