mlsanityData to 7 Oct 2026

Offline Inverse Reinforcement Learning

arXiv 2106.05068 ↗First posted 9 Jun 2021cs.LGcited by 14

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

The objective of offline RL is to learn optimal policies when a fixed exploratory demonstrations data-set is available and sampling additional observations is impossible (typically if this operation is either costly or rises ethical questions). In order to solve this problem, off the shelf approaches require a properly defined cost function (or its evaluation on the provided data-set), which are seldom available in practice. To circumvent this issue, a reasonable alternative is to query an expert for few optimal demonstrations in addition to the exploratory data-set. The objective is then to learn an optimal policy w.r.t. the expert's latent cost function. Current solutions either solve a behaviour cloning problem (which does not leverage the exploratory data) or a reinforced imitation learning problem (using a fixed cost function that discriminates available exploratory trajectories from expert ones). Inspired by the success of IRL techniques in achieving state of the art imitation performances in online settings, we exploit GAN based data augmentation procedures to construct the first offline IRL algorithm. The obtained policies outperformed the aforementioned solutions on multiple OpenAI gym environments.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 33 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

23 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021A Generalised Inverse Reinforcement Learning Framework25 May
  2. 2020Regularized Inverse Reinforcement Learning7 Oct
  3. 2018Adversarial Imitation via Variational Inverse Reinforcement Learning17 Sep

Other references · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2014Conditional Generative Adversarial Netscited by 2,152
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  12. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
8 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  2. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  3. 2019A Theory of Regularized Markov Decision Processescited by 279
  4. 2018Learning from positive and unlabeled data: a surveycited by 218
  5. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  6. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  7. 2020Semi-supervised reward learning for offline reinforcement learningcited by 22
  8. 2019Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 4; 2025: 1; 2026: 2.

14 papers.

Sort:

By area:

  1. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  2. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
  3. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  4. 2024Learning from Imperfect Demonstrations with Self-Supervision for Robotic Manipulationcited by 13
  5. 2023CEIL: Generalized Contextual Imitation Learningcited by 12
  6. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  7. 2024Offline Imitation Learning with Model-based Reverse Augmentationcited by 8
  8. 2023Contrastive Example-Based Controlcited by 3
  9. 2024Offline Reinforcement Learning with Imputed Rewardscited by 1
  10. 2025OffSim: Offline Simulator for Model-based Offline Inverse Reinforcement Learningcited by 1
  11. 2026Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignmentcited by 1
  12. 2023On Combining Expert Demonstrations in Imitation Learning via Optimal Transportcited by 0
2 more
  1. 2024The Virtues of Pessimism in Inverse Reinforcement Learningcited by 0
  2. 2026LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decompositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms
  2. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  4. 2020Off-Policy Adversarial Inverse Reinforcement Learning
  5. 2024SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning
  6. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  7. 2021A Minimalist Approach to Offline Reinforcement Learning
  8. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning