# oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions

arXiv: https://arxiv.org/abs/2002.09043 · HTML: https://mlsanity.com/paper/2002.09043/

## Key facts

- Cited by 3 papers in the map (data through 7 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 20 Feb 2020.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 16 of 16)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Reverse Curriculum Generation for Reinforcement Learning](https://mlsanity.com/paper/1707.05300/) (2017)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Robust Imitation of Diverse Behaviors](https://mlsanity.com/paper/1707.02747/) (2017)
- [Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets](https://mlsanity.com/paper/1705.10479/) (2017)
- [Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information](https://mlsanity.com/paper/1810.01266/) (2018)
- [OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1709.06683/) (2017)
- [Metropolis-Hastings Generative Adversarial Networks](https://mlsanity.com/paper/1811.11357/) (2018)
- [Learnings Options End-to-End for Continuous Action Tasks](https://mlsanity.com/paper/1712.00004/) (2017)
- [HIRL: Hierarchical Inverse Reinforcement Learning for Long-Horizon Tasks with Delayed Rewards](https://mlsanity.com/paper/1604.06508/) (2016)

## Built on by (most cited first, 3 of 3)

- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions](https://mlsanity.com/paper/2402.13777/) (2024)
- [Context-Hierarchy Inverse Reinforcement Learning](https://mlsanity.com/paper/2202.12597/) (2022)

## Abstract (arXiv)

Explicit engineering of reward functions for given environments has been a major hindrance to reinforcement learning methods. While Inverse Reinforcement Learning (IRL) is a solution to recover reward functions from demonstrations only, these learned rewards are generally heavily textitentangled with the dynamics of the environment and therefore not portable or robust to changing environments. Modern adversarial methods have yielded some success in reducing reward entanglement in the IRL setting. In this work, we leverage one such method, Adversarial Inverse Reinforcement Learning (AIRL), to propose an algorithm that learns hierarchical disentangled rewards with a policy over options. We show that this method has the ability to learn generalizable policies and reward functions in complex transfer learning tasks, while yielding results in continuous control benchmarks that are comparable to those of the state-of-the-art methods.