mlsanityData to 7 Oct 2026

Hierarchical Imitation and Reinforcement Learning

arXiv 1803.00590 ↗First posted 1 Mar 2018cs.LG cs.AI stat.MLcited by 88

Not on the map. Why?

Cited by 47 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study how to effectively leverage expert feedback to learn sequential decision-making policies. We focus on problems with sparse rewards and long time horizons, which typically pose significant challenges in reinforcement learning. We propose an algorithmic framework, called hierarchical guidance, that leverages the hierarchical structure of the underlying problem to integrate different modes of expert interaction. Our framework can incorporate different combinations of imitation learning (IL) and reinforcement learning (RL) at different levels, leading to dramatic reductions in both expert effort and cost of exploration. Using long-horizon benchmarks, including Montezuma's Revenge, we demonstrate that our approach can learn significantly faster than hierarchical RL, and be significantly more label-efficient than standard IL. We also theoretically analyze labeling cost for certain instantiations of our framework.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 4; 2019: 2; 2020: 3; 2021: 15; 2022: 19; 2023: 18; 2024: 12; 2025: 7; 2026: 8.

88 papers.

Sort:

By area:

  1. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  2. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulationcited by 82
  3. 2022BITS: Bi-level Imitation for Traffic Simulationcited by 80
  4. 2023XSkill: Cross Embodiment Skill Discoverycited by 74
  5. 2023LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discoverycited by 63
  6. 2023Multi-Stage Cable Routing through Hierarchical Imitation Learningcited by 59
  7. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
  8. 2020Deep Imitation Learning for Bimanual Robotic Manipulationcited by 49
  9. 2021From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligencecited by 35
  10. 2022Chain of Thought Imitation with Procedure Cloningcited by 30
  11. 2021JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learningcited by 28
  12. 2023Human-in-the-Loop Task and Motion Planning for Imitation Learningcited by 28
38 more
  1. 2023Second Thoughts are Best: Learning to Re-Align With Human Values from Text Editscited by 27
  2. 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Rewardcited by 26
  3. 2024Autoregressive Action Sequence Learning for Robotic Manipulationcited by 26
  4. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  5. 2025Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewardscited by 23
  6. 2021Continuous Control with Action Quantization from Demonstrationscited by 21
  7. 2024PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Controlcited by 21
  8. 2021Adversarial Option-Aware Hierarchical Imitation Learningcited by 19
  9. 2022Policy Optimization with Linear Temporal Logic Constraintscited by 19
  10. 2018Imitation Learning with Concurrent Actions in 3D Gamescited by 18
  11. 2021Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learningcited by 17
  12. 2022Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Controlcited by 16
  13. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learningcited by 15
  14. 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
  15. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13
  16. 2022Conditional Imitation Learning for Multi-Agent Gamescited by 13
  17. 2025Think Small, Act Big: Primitive Prompt Learning for Lifelong Robot Manipulationcited by 13
  18. 2021A Framework for Learning to Request Rich and Contextually Useful Information from Humanscited by 12
  19. 2023NetHack is Hard to Hackcited by 11
  20. 2020Provable Hierarchical Imitation Learning via EMcited by 10
  21. 2022A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with Symbolic Reward Machinescited by 9
  22. 2023LLM Augmented Hierarchical Agentscited by 9
  23. 2021Interpretable Model-based Hierarchical Reinforcement Learning using Inductive Logic Programmingcited by 8
  24. 2019Integration of Imitation Learning using GAIL and Reinforcement Learning using Task-achievement Rewards via Probabilistic Graphical Modelcited by 7
  25. 2021OnSlicing: Online End-to-End Network Slicing with Reinforcement Learningcited by 7
  26. 2022Transfering Hierarchical Structure with Dual Meta Imitation Learningcited by 7
  27. 2022Goal-Aware Generative Adversarial Imitation Learning from Imperfect Demonstration for Robotic Cloth Manipulationcited by 7
  28. 2022Model-based trajectory stitching for improved behavioural cloning and its applicationscited by 7
  29. 2018Accelerating Imitation Learning with Predictive Modelscited by 6
  30. 2022Learning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstrationscited by 6
  31. 2023Conceptual Framework for Autonomous Cognitive Entitiescited by 6
  32. 2023GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimationcited by 6
  33. 2022SHAIL: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environmentscited by 5
  34. 2022Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gapcited by 5
  35. 2024IDIL: Imitation Learning of Intent-Driven Expert Behaviorcited by 5
  36. 2023Disturbance Injection under Partial Automation: Robust Imitation Learning for Long-horizon Taskscited by 4
  37. 2024Feudal Networks for Visual Navigationcited by 4
  38. 2024Offline Hierarchical Reinforcement Learning via Inverse Optimizationcited by 4

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learning
  2. 2019Reinforced Imitation in Heterogeneous Action Space
  3. 2022Hierarchical Decision Transformer
  4. 2020Active Hierarchical Imitation and Reinforcement Learning
  5. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning
  6. 2018Fast Policy Learning through Imitation and Reinforcement
  7. 2019Reward-Conditioned Policies
  8. 2024MENTOR: Guiding Hierarchical Reinforcement Learning with Human Feedback and Dynamic Distance Constraint