mlsanityData to 7 Oct 2026

Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control

arXiv 2210.01969 ↗First posted 5 Oct 2022cs.LG cs.AI cs.ROcited by 16

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Hierarchical Imitation Learning (HIL) has been proposed to recover highly-complex behaviors in long-horizon tasks from expert demonstrations by modeling the task hierarchy with the option framework. Existing methods either overlook the causal relationship between the subtask and its corresponding policy or cannot learn the policy in an end-to-end fashion, which leads to suboptimality. In this work, we develop a novel HIL algorithm based on Adversarial Inverse Reinforcement Learning and adapt it with the Expectation-Maximization algorithm in order to directly recover a hierarchical policy from the unannotated demonstrations. Further, we introduce a directed information term to the objective function to enhance the causality and propose a Variational Autoencoder framework for learning with our objectives in an end-to-end fashion. Theoretical justifications and evaluations on challenging robotic control tasks are provided to show the superiority of our algorithm. The codes are available at https://github.com/LucasCJYSDL/HierAIRL.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 18 of its 43 references that are in the map. First posted on arXiv 5 Oct 2022.

Builds on

18 of 43 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2016OpenAI Gymcited by 1,684
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  9. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  10. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  11. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  12. 2019Stacked Capsule Autoencoderscited by 82
6 more
  1. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  2. 2018Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Informationcited by 41
  3. 2016Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replaycited by 22
  4. 2020Scalable Multi-Task Imitation Learning with Autonomous Improvementcited by 21
  5. 2021Adversarial Option-Aware Hierarchical Imitation Learningcited by 19
  6. 2020Provable Hierarchical Imitation Learning via EMcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 2; 2025: 4; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
  2. 2023Multi-task Hierarchical Adversarial Inverse Reinforcement Learningcited by 15
  3. 2023Multi-intention Inverse Q-learning for Interpretable Behavior Representationcited by 9
  4. 2025Global Convergence for Average Reward Constrained MDPs with Primal-Dual Actor Critic Algorithmcited by 8
  5. 2022A Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Processcited by 7
  6. 2023GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimationcited by 6
  7. 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversariescited by 6
  8. 2025Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviorscited by 6
  9. 2024IDIL: Imitation Learning of Intent-Driven Expert Behaviorcited by 5
  10. 2023Hierarchical Deep Counterfactual Regret Minimizationcited by 4
  11. 2025Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithmcited by 1
  12. 2025Primal-Only Actor Critic Algorithm for Robust Constrained Average Cost MDPscited by 1
4 more
  1. 2023Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learningcited by 0
  2. 2023CasIL: Cognizing and Imitating Skills via a Dual Cognition-Action Architecturecited by 0
  3. 2026E2DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulationcited by 0
  4. 2026Hierarchical Multilevel Monte Carlo for Order-Optimal Neural Actor-Critic in Average-Reward CMDPscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020NEARL: Non-Explicit Action Reinforcement Learning for Robotic Control
  2. 2018Adversarial Imitation via Variational Inverse Reinforcement Learning
  3. 2021Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks
  4. 2020Off-Policy Adversarial Inverse Reinforcement Learning
  5. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment
  6. 2020oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions
  7. 2025SPLASH! Sample-efficient Preference-based inverse reinforcement learning for Long-horizon Adversarial tasks from Suboptimal Hierarchical demonstrations
  8. 2019Multi-Agent Adversarial Inverse Reinforcement Learning