mlsanityData to 7 Oct 2026

Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

arXiv 1909.10618 ↗First posted 23 Sep 2019cs.LG cs.AI stat.MLcited by 66

Not on the map. Why?

Abstract (arXiv)

Hierarchical reinforcement learning has demonstrated significant success at solving difficult reinforcement learning (RL) tasks. Previous works have motivated the use of hierarchy by appealing to a number of intuitive benefits, including learning over temporally extended transitions, exploring over temporally extended periods, and training and exploring in a more semantically meaningful action space, among others. However, in fully observed, Markovian settings, it is not immediately clear why hierarchical RL should provide benefits over standard "shallow" RL architectures. In this work, we isolate and evaluate the claimed benefits of hierarchical RL on a suite of tasks encompassing locomotion, navigation, and manipulation. Surprisingly, we find that most of the observed benefits of hierarchy can be attributed to improved exploration, as opposed to easier policy learning or imposed hierarchical structures. Given this insight, we present exploration techniques inspired by hierarchy that achieve performance competitive with hierarchical RL while at the same time being much simpler to use and implement.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 11; 2022: 15; 2023: 8; 2024: 12; 2025: 10; 2026: 9.

66 papers.

Sort:

By area:

  1. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  2. 2024Simple Hierarchical Planning with Diffusioncited by 71
  3. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
  4. 2025Horizon Reduction Makes RL Scalablecited by 55
  5. 2022Skill-based Model-based Reinforcement Learningcited by 50
  6. 2021Hierarchical Skills for Efficient Explorationcited by 37
  7. 2024Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robotscited by 32
  8. 2021Discovery of Options via Meta-Learned Subgoalscited by 27
  9. 2021Learning Transferable Motor Skills with Hierarchical Latent Mixture Policiescited by 26
  10. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25
  11. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 25
  12. 2024Hierarchical World Models as Visual Whole-Body Humanoid Controllerscited by 24
38 more
  1. 2022Causality-driven Hierarchical Structure Discovery for Reinforcement Learningcited by 23
  2. 2021Active Hierarchical Exploration with Stable Subgoal Representation Learningcited by 17
  3. 2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
  4. 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
  5. 2025Horizon Generalization in Reinforcement Learningcited by 15
  6. 2025Efficient Robotic Policy Learning via Latent Space Backward Planningcited by 14
  7. 2021Adjacency constraint for efficient hierarchical reinforcement learningcited by 10
  8. 2022Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learningcited by 10
  9. 2022Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Taskscited by 10
  10. 2023Hieros: Hierarchical Imagination on Structured State Space Sequence World Modelscited by 8
  11. 2025Flattening Hierarchies with Policy Bootstrappingcited by 8
  12. 2023Natural Language-conditioned Reinforcement Learning with Inside-out Task Language Development and Translationcited by 7
  13. 2022SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Explorationcited by 6
  14. 2021Provable Hierarchy-Based Meta-Reinforcement Learningcited by 5
  15. 2022Learning Minimally-Violating Continuous Control for Infeasible Linear Temporal Logic Specificationscited by 5
  16. 2022Challenges to Solving Combinatorially Hard Long-Horizon Deep RL Taskscited by 4
  17. 2023Hierarchically Composing Level Generators for the Creation of Complex Structurescited by 4
  18. 2024Feudal Networks for Visual Navigationcited by 4
  19. 2021Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learningcited by 3
  20. 2022Developing cooperative policies for multi-stage reinforcement learning taskscited by 3
  21. 2024PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabelingcited by 3
  22. 2022From Scratch to Sketch: Deep Decoupled Hierarchical Reinforcement Learning for Robotic Sketching Agentcited by 2
  23. 2023CRISP: Curriculum Inducing Primitive Informed Subgoal Prediction for Hierarchical Reinforcement Learningcited by 2
  24. 2023PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learningcited by 2
  25. 2025TAG: A Decentralized Framework for Multi-Agent Hierarchical Reinforcement Learningcited by 2
  26. 2026Quantum Hierarchical Reinforcement Learning via Variational Quantum Circuitscited by 2
  27. 2023Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphscited by 1
  28. 2024DIPPER: Direct Preference Optimization to Accelerate Primitive-Enabled Hierarchical Reinforcement Learningcited by 1
  29. 2024Efficient Imitation Without Demonstrations via Value-Penalized Auxiliary Control from Examplescited by 1
  30. 2024Hierarchical Average-Reward Linearly-solvable Markov Decision Processescited by 1
  31. 2024Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approachcited by 1
  32. 2025Hierarchical Reinforcement Learning in Multi-Goal Spatial Navigation with Autonomous Mobile Robotscited by 1
  33. 2025Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learningcited by 1
  34. 2026Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoningcited by 1
  35. 2021TradeR: Practical Deep Hierarchical Reinforcement Learning for Trade Executioncited by 0
  36. 2022Exploiting Semantic Epsilon Greedy Exploration Strategy in Multi-Agent Reinforcement Learningcited by 0
  37. 2022SAGE: Generating Symbolic Goals for Myopic Models in Deep Reinforcement Learningcited by 0
  38. 2022Emergency action termination for immediate reaction in hierarchical reinforcement learningcited by 0

50 most cited of 66 All 66 (JSON)