mlsanityData to 14 Oct 2026

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

arXiv 1604.06057 ↗First posted 20 Apr 2016cs.LG cs.AI cs.CVcited by 286

Not on the map. Why?

Cited by 65 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. The primary difficulty arises due to insufficient exploration, resulting in an agent being unable to learn robust value functions. Intrinsically motivated agents can explore new behavior for its own sake rather than to directly solve problems. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchical-DQN (h-DQN), a framework to integrate hierarchical value functions, operating at different temporal scales, with intrinsically motivated deep reinforcement learning. A top-level value function learns a policy over intrinsic goals, and a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse, delayed feedback: (1) a complex discrete stochastic decision process, and (2) the classic ATARI game `Montezuma's Revenge'.

Key facts. Cited by 286 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Apr 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 3; 2019: 0; 2020: 3; 2021: 55; 2022: 58; 2023: 43; 2024: 40; 2025: 46; 2026: 38.

286 papers.

Sort:

By area:

  1. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  2. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  3. 2025Reinforcement Learning with Action Chunkingcited by 107
  4. 2021Skill Induction and Planning with Latent Languagecited by 99
  5. 2023Modular Deep Learningcited by 83
  6. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  7. 2023Collaborating with language models for embodied reasoningcited by 61
  8. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 61
  9. 2021A Survey of Machine Learning for Computer Architecture and Systemscited by 53
  10. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 49
  11. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 47
  12. 2021Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularizationcited by 46
38 more
  1. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  2. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 36
  3. 2022MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffercited by 34
  4. 2021LASER: Learning a Latent Action Space for Efficient Reinforcement Learningcited by 32
  5. 2021Discovery of Options via Meta-Learned Subgoalscited by 27
  6. 2021Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learningcited by 27
  7. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 26
  8. 2021Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Surveycited by 24
  9. 2021Reinforcement Learning for IoT Security: A Comprehensive Surveycited by 23
  10. 2022AI Autonomy : Self-Initiated Open-World Continual Learning and Adaptationcited by 23
  11. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 23
  12. 2022Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversitycited by 22
  13. 2022Learning Options via Compressioncited by 22
  14. 2024Empowering Large Language Model Agents through Action Learningcited by 22
  15. 2021A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directionscited by 21
  16. 2022Intelligent problem-solving as integrated hierarchical reinforcement learningcited by 21
  17. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimizationcited by 20
  18. 2021Solving Compositional Reinforcement Learning Problems via Task Reductioncited by 19
  19. 2022ALMA: Hierarchical Learning for Composite Multi-Agent Taskscited by 18
  20. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  21. 2026HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agentscited by 18
  22. 2021Active Hierarchical Exploration with Stable Subgoal Representation Learningcited by 17
  23. 2021Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learningcited by 17
  24. 2021How and Why to Manipulate Your Own Agent: On the Incentives of Users of Learning Agentscited by 17
  25. 2025Horizon Generalization in Reinforcement Learningcited by 17
  26. 2022GALOIS: Boosting Deep Reinforcement Learning via Generalizable Logic Synthesiscited by 16
  27. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16
  28. 2022Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learningcited by 16
  29. 2022Reinforcement Learning in Medical Image Analysis: Concepts, Applications, Challenges, and Future Directionscited by 16
  30. 2023Generalizable Long-Horizon Manipulations with Large Language Modelscited by 16
  31. 2022DANLI: Deliberative Agent for Following Natural Language Instructionscited by 15
  32. 2024Hierarchical State Space Models for Continuous Sequence-to-Sequence Modelingcited by 15
  33. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 15
  34. 2025LLM-CAS: Dynamic Neuron Perturbation for Real-Time Hallucination Correctioncited by 15
  35. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 14
  36. 2021Verifiable and Compositional Reinforcement Learning Systemscited by 13
  37. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13
  38. 2022Generalised Policy Improvement with Geometric Policy Compositioncited by 13

50 most cited of 286 All 286 (JSON)