mlsanityData to 7 Oct 2026

The Virtues of Pessimism in Inverse Reinforcement Learning

arXiv 2402.02616 ↗First posted 4 Feb 2024cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse Reinforcement Learning (IRL) is a powerful framework for learning complex behaviors from expert demonstrations. However, it traditionally requires repeatedly solving a computationally expensive reinforcement learning (RL) problem in its inner loop. It is desirable to reduce the exploration burden by leveraging expert demonstrations in the inner-loop RL. As an example, recent work resets the learner to expert states in order to inform the learner of high-reward expert states. However, such an approach is infeasible in the real world. In this work, we consider an alternative approach to speeding up the RL subroutine in IRL: pessimism, i.e., staying close to the expert's data distribution, instantiated via the use of offline RL algorithms. We formalize a connection between offline RL and IRL, enabling us to use an arbitrary offline RL algorithm to improve the sample efficiency of IRL. We validate our theory experimentally by demonstrating a strong correlation between the efficacy of an offline RL algorithm and how well it works as part of an IRL procedure. By using a strong offline RL algorithm as part of an IRL procedure, we are able to find policies that match expert performance significantly more efficiently than the prior art.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 41 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

27 of 41 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Inverse Reinforcement Learning without Reinforcement Learning26 Mar
  2. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning9 Feb
  3. 2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching9 Sep
  4. 2021Offline Inverse Reinforcement Learning9 Jun
  5. 2021Scalable Bayesian Inverse Reinforcement Learning12 Feb
  6. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov

Other references · 21

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  9. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  10. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  11. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  12. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
9 more
  1. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  2. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  3. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  4. 2022Causal Imitation Learning with Unobserved Confounderscited by 40
  5. 2022Sequence Model Imitation Learning with Unobserved Contextscited by 34
  6. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewardscited by 28
  7. 2022Causal Imitation Learning under Temporally Correlated Noisecited by 28
  8. 2022Minimax Optimal Online Imitation Learning via Replay Estimationcited by 25
  9. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  2. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms
  3. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  4. 2022Active Exploration for Inverse Reinforcement Learning
  5. 2024Hybrid Inverse Reinforcement Learning
  6. 2020Is Pessimism Provably Efficient for Offline RL?
  7. 2022Receding Horizon Inverse Reinforcement Learning
  8. 2023A Bayesian Approach to Robust Inverse Reinforcement Learning