mlsanityData to 7 Oct 2026

Option Compatible Reward Inverse Reinforcement Learning

arXiv 1911.02723 ↗First posted 7 Nov 2019cs.LG stat.MLcited by 2

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Reinforcement learning in complex environments is a challenging problem. In particular, the success of reinforcement learning algorithms depends on a well-designed reward function. Inverse reinforcement learning (IRL) solves the problem of recovering reward functions from expert demonstrations. In this paper, we solve a hierarchical inverse reinforcement learning problem within the options framework, which allows us to utilize intrinsic motivation of the expert demonstrations. A gradient method for parametrized options is used to deduce a defining equation for the Q-feature space, which leads to a reward feature space. Using a second-order optimality condition for option parameters, an optimal reward function is selected. Experimental results in both discrete and continuous domains confirm that our recovered rewards provide a solution to the IRL problem using temporal abstraction, which in turn are effective in accelerating transfer learning tasks. We also show that our method is robust to noises contained in expert demonstrations.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 8 of its 27 references that are in the map. First posted on arXiv 7 Nov 2019.

Builds on

8 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress18 Jun

Other references · 7

  1. 2016The Option-Critic Architecturecited by 433
  2. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  3. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
  4. 2017Multi-Level Discovery of Deep Optionscited by 50
  5. 2017OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learningcited by 40
  6. 2016HIRL: Hierarchical Inverse Reinforcement Learning for Long-Horizon Tasks with Delayed Rewardscited by 15
  7. 2017Composing Meta-Policies for Autonomous Driving Using Hierarchical Deep Reinforcement Learningcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 2.

2 papers.

  1. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
  2. 2022Context-Hierarchy Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
  2. 2024Automated Feature Selection for Inverse Reinforcement Learning
  3. 2025Reward Compatibility: A Framework for Inverse RL
  4. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  5. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization
  6. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
  7. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning
  8. 2021Versatile Inverse Reinforcement Learning via Cumulative Rewards