mlsanityData to 7 Oct 2026

Inverse Reinforcement Learning without Reinforcement Learning

arXiv 2303.14623 ↗First posted 26 Mar 2023cs.LGcited by 35

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse Reinforcement Learning (IRL) is a powerful set of techniques for imitation learning that aims to learn a reward function that rationalizes expert demonstrations. Unfortunately, traditional IRL methods suffer from a computational weakness: they require repeatedly solving a hard reinforcement learning (RL) problem as a subroutine. This is counter-intuitive from the viewpoint of reductions: we have reduced the easier problem of imitation learning to repeatedly solving the harder problem of RL. Another thread of work has proved that access to the side-information of the distribution of states where a strong policy spends time can dramatically reduce the sample and computational complexities of solving an RL problem. In this work, we demonstrate for the first time a more informed imitation learning reduction where we utilize the state distribution of the expert to alleviate the global exploration component of the RL subroutine, providing an exponential speedup in theory. In practice, we find that we are able to significantly speed up the prior art on continuous control tasks.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 57 references that are in the map. First posted on arXiv 26 Mar 2023.

Builds on

21 of 57 references.

References · 21

  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2017Improved Training of Wasserstein GANscited by 2,223
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  8. 2019Introduction to Online Convex Optimizationcited by 793
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  10. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  11. 2017Training GANs with Optimismcited by 217
  12. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
9 more
  1. 2022Jump-Start Reinforcement Learningcited by 102
  2. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  3. 2019Provably Efficient Imitation Learning from Observation Alonecited by 78
  4. 2022Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulationcited by 61
  5. 2022Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real worldcited by 56
  6. 2015Learning to Search Better Than Your Teachercited by 50
  7. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  8. 2022Hierarchical Model-Based Imitation Learning for Planning in Autonomous Drivingcited by 43
  9. 2022Minimax Optimal Online Imitation Learning via Replay Estimationcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 17; 2025: 11; 2026: 5.

35 papers.

Sort:

By area:

  1. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  2. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  3. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 51
  4. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  5. 2024Diffusion-Reward Adversarial Imitation Learningcited by 18
  6. 2024Diffusing States and Matching Scores: A New Framework for Imitation Learningcited by 16
  7. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 16
  8. 2023Learning Shared Safety Constraints from Multi-task Demonstrationscited by 15
  9. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptionscited by 15
  10. 2025Perception in Reflectioncited by 13
  11. 2023Massively Scalable Inverse Reinforcement Learning in Google Mapscited by 11
  12. 2024Multi-Agent Imitation Learning: Value is Easy, Regret is Hardcited by 11
23 more
  1. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  2. 2024In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminatescited by 7
  3. 2024Provably and Practically Efficient Adversarial Imitation Learning with General Function Approximationcited by 6
  4. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  5. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  6. 2025Efficient Imitation under Misspecificationcited by 4
  7. 2025Learning Equilibria from Data: Provably Efficient Multi-Agent Imitation Learningcited by 4
  8. 2025Best Policy Learning from Trajectory Preference Feedbackcited by 3
  9. 2024Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understandingcited by 2
  10. 2025Your Learned Constraint is Secretly a Backward Reachable Tubecited by 2
  11. 2024NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fieldscited by 1
  12. 2025Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithmcited by 1
  13. 2026Multi-agent imitation learning with function approximation: Linear Markov games and beyondcited by 1
  14. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithmscited by 1
  15. 2024The Virtues of Pessimism in Inverse Reinforcement Learningcited by 0
  16. 2024Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proofcited by 0
  17. 2024Unlocking the Potential of Global Human Expertisecited by 0
  18. 2024Improving Trust Estimation in Human-Robot Collaboration Using Beta Reputation at Fine-grained Timescalescited by 0
  19. 2025Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advancescited by 0
  20. 2025Going Beyond Expert Performance via Deep Implicit Imitation Reinforcement Learningcited by 0
  21. 2026Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanismcited by 0
  22. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learningcited by 0
  23. 2026Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  2. 2024Hybrid Inverse Reinforcement Learning
  3. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms
  4. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  5. 2021Inverse Reinforcement Learning in a Continuous State Space with Formal Guarantees
  6. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstrators
  7. 2023Curricular Subgoals for Inverse Reinforcement Learning
  8. 2024Accelerating Inverse Reinforcement Learning with Expert Bootstrapping