# Offline Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2106.05068 · HTML: https://mlsanity.com/paper/2106.05068/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 23 of 23)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021)
- [A Theory of Regularized Markov Decision Processes](https://mlsanity.com/paper/1901.11275/) (2019)
- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [Learning from positive and unlabeled data: a survey](https://mlsanity.com/paper/1811.04820/) (2018)
- [C-Learning: Learning to Achieve Goals via Recursive Classification](https://mlsanity.com/paper/2011.08909/) (2020)
- [Offline Learning from Demonstrations and Unlabeled Experience](https://mlsanity.com/paper/2011.13885/) (2020)
- [Adversarial Imitation via Variational Inverse Reinforcement Learning](https://mlsanity.com/paper/1809.06404/) (2018)
- [Semi-supervised reward learning for offline reinforcement learning](https://mlsanity.com/paper/2012.06899/) (2020)
- [Regularized Inverse Reinforcement Learning](https://mlsanity.com/paper/2010.03691/) (2020)
- [Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning](https://mlsanity.com/paper/1911.00238/) (2019)
- [A Generalised Inverse Reinforcement Learning Framework](https://mlsanity.com/paper/2105.11812/) (2021)

## Built on by (most cited first, 14 of 14)

- [Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations](https://mlsanity.com/paper/2207.10050/) (2022)
- [Real World Offline Reinforcement Learning with Realistic Data Source](https://mlsanity.com/paper/2210.06479/) (2022)
- [Mind the Gap: Offline Policy Optimization for Imperfect Rewards](https://mlsanity.com/paper/2302.01667/) (2023)
- [Learning from Imperfect Demonstrations with Self-Supervision for Robotic Manipulation](https://mlsanity.com/paper/2401.08957/) (2024)
- [CEIL: Generalized Contextual Imitation Learning](https://mlsanity.com/paper/2306.14534/) (2023)
- [CLUE: Calibrated Latent Guidance for Offline Reinforcement Learning](https://mlsanity.com/paper/2306.13412/) (2023)
- [Offline Imitation Learning with Model-based Reverse Augmentation](https://mlsanity.com/paper/2406.12550/) (2024)
- [Contrastive Example-Based Control](https://mlsanity.com/paper/2307.13101/) (2023)
- [Offline Reinforcement Learning with Imputed Rewards](https://mlsanity.com/paper/2407.10839/) (2024)
- [On Combining Expert Demonstrations in Imitation Learning via Optimal Transport](https://mlsanity.com/paper/2307.10810/) (2023)
- [OffSim: Offline Simulator for Model-based Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2510.15495/) (2025)
- [Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignment](https://mlsanity.com/paper/2602.14844/) (2026)
- [The Virtues of Pessimism in Inverse Reinforcement Learning](https://mlsanity.com/paper/2402.02616/) (2024)
- [LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decomposition](https://mlsanity.com/paper/2605.24005/) (2026)

## Abstract (arXiv)

The objective of offline RL is to learn optimal policies when a fixed exploratory demonstrations data-set is available and sampling additional observations is impossible (typically if this operation is either costly or rises ethical questions). In order to solve this problem, off the shelf approaches require a properly defined cost function (or its evaluation on the provided data-set), which are seldom available in practice. To circumvent this issue, a reasonable alternative is to query an expert for few optimal demonstrations in addition to the exploratory data-set. The objective is then to learn an optimal policy w.r.t. the expert's latent cost function. Current solutions either solve a behaviour cloning problem (which does not leverage the exploratory data) or a reinforced imitation learning problem (using a fixed cost function that discriminates available exploratory trajectories from expert ones). Inspired by the success of IRL techniques in achieving state of the art imitation performances in online settings, we exploit GAN based data augmentation procedures to construct the first offline IRL algorithm. The obtained policies outperformed the aforementioned solutions on multiple OpenAI gym environments.