mlsanityData to 7 Oct 2026

Data-Efficient Hierarchical Reinforcement Learning

arXiv 1805.08296 ↗First posted 21 May 2018cs.LG cs.AI stat.MLcited by 436

Not on the map. Why?

Cited by 53 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Hierarchical reinforcement learning (HRL) is a promising approach to extend traditional reinforcement learning (RL) methods to solve more complex tasks. Yet, the majority of current HRL methods require careful task-specific design and on-policy training, making them difficult to apply in real-world scenarios. In this paper, we study how we can develop HRL algorithms that are general, in that they do not make onerous additional assumptions beyond standard RL algorithms, and efficient, in the sense that they can be used with modest numbers of interaction samples, making them suitable for real-world problems such as robotic control. For generality, we develop a scheme where lower-level controllers are supervised with goals that are learned and proposed automatically by the higher-level controllers. To address efficiency, we propose to use off-policy experience for both higher and lower-level training. This poses a considerable challenge, since changes to the lower-level behaviors change the action space for the higher-level policy, and we introduce an off-policy correction to remedy this challenge. This allows us to take advantage of recent advances in off-policy model-free RL to learn both higher- and lower-level policies using substantially fewer environment interactions than on-policy algorithms. We term the resulting HRL agent HIRO and find that it is generally applicable and highly sample-efficient. Our experiments show that HIRO can be used to learn highly complex behaviors for simulated robots, such as pushing objects and utilizing them to reach target locations, learning from only a few million samples, equivalent to a few days of real-time interaction. In comparisons with a number of prior HRL methods, we find that our approach substantially outperforms previous state-of-the-art techniques.

Key facts. Cited by 436 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 4; 2021: 72; 2022: 83; 2023: 66; 2024: 69; 2025: 66; 2026: 75.

436 papers.

Sort:

By area:

  1. 2023Learning Interactive Real-World Simulatorscited by 440
  2. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  3. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  4. 2021The Values Encoded in Machine Learning Researchcited by 172
  5. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  6. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  7. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  8. 2023Large Language Models for Robotics: A Surveycited by 132
  9. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  10. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  11. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  12. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101
38 more
  1. 2025Reinforcement Learning with Action Chunkingcited by 97
  2. 2022Deep Hierarchical Planning from Pixelscited by 96
  3. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
  4. 2023Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Modelscited by 87
  5. 2023Modular Deep Learningcited by 82
  6. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  7. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
  8. 2019State Alignment-based Imitation Learningcited by 73
  9. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  10. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Executioncited by 69
  11. 2023SteP: Stacked LLM Policies for Web Actionscited by 66
  12. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 63
  13. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
  14. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  15. 2022Learning Neuro-Symbolic Skills for Bilevel Planningcited by 55
  16. 2025Horizon Reduction Makes RL Scalablecited by 55
  17. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
  18. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 51
  19. 2022Skill-based Model-based Reinforcement Learningcited by 50
  20. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  21. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
  22. 2023Towards A Unified Agent with Foundation Modelscited by 46
  23. 2021Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularizationcited by 44
  24. 2023Learning Cut Selection for Mixed-Integer Linear Programming via Hierarchical Sequence Modelcited by 44
  25. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 43
  26. 2024COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Modelscited by 43
  27. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  28. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  29. 2021Hierarchical Skills for Efficient Explorationcited by 37
  30. 2021From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligencecited by 35
  31. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  32. 2023Contrastive Difference Predictive Codingcited by 35
  33. 2022LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulationcited by 32
  34. 2024Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robotscited by 32
  35. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environmentscited by 32
  36. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  37. 2022LISA: Learning Interpretable Skill Abstractions from Languagecited by 30
  38. 2023Self-optimizing Feature Generation via Categorical Hashing Representation and Hierarchical Reinforcement Crossingcited by 30

50 most cited of 436 All 436 (JSON)