mlsanityData to 14 Oct 2026

Intelligent problem-solving as integrated hierarchical reinforcement learning

arXiv 2208.08731 ↗First posted 18 Aug 2022cs.AI cs.LG cs.ROcited by 21

Not on the map. Why?

Abstract (arXiv)

According to cognitive psychology and related disciplines, the development of complex problem-solving behaviour in biological agents depends on hierarchical cognitive mechanisms. Hierarchical reinforcement learning is a promising computational approach that may eventually yield comparable problem-solving behaviour in artificial agents and robots. However, to date the problem-solving abilities of many human and non-human animals are clearly superior to those of artificial systems. Here, we propose steps to integrate biologically inspired hierarchical mechanisms to enable advanced problem-solving skills in artificial agents. Therefore, we first review the literature in cognitive psychology to highlight the importance of compositional abstraction and predictive processing. Then we relate the gained insights with contemporary hierarchical reinforcement learning methods. Interestingly, our results suggest that all identified cognitive mechanisms have been implemented individually in isolated computational architectures, raising the question of why there exists no single unifying architecture that integrates them. As our final contribution, we address this question by providing an integrative perspective on the computational challenges to develop such a unifying architecture. We expect our results to guide the development of more sophisticated cognitively inspired hierarchical machine learning architectures.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 43 of its 134 references that are in the map. First posted on arXiv 18 Aug 2022.

Builds on

43 of 134 references.

References · 43

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  3. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  8. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  9. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  10. 2016The Option-Critic Architecturecited by 448
  11. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  12. 2017Deep Reinforcement Learning: An Overviewcited by 333
31 more
  1. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  2. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  3. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  4. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  5. 2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 127
  6. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
  7. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 98
  8. 2016Playing FPS Games with Deep Reinforcement Learningcited by 95
  9. 2017Meta Learning Shared Hierarchiescited by 93
  10. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  11. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 79
  12. 2018SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planningcited by 78
  13. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67
  14. 2016Learning and Transfer of Modulated Locomotor Controllerscited by 59
  15. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  16. 2020On the role of planning in model-based deep reinforcement learningcited by 57
  17. 2018PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Makingcited by 52
  18. 2019Sub-policy Adaptation for Hierarchical Reinforcement Learningcited by 47
  19. 2018Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependenciescited by 43
  20. 2016Strategic Attentive Writer for Learning Macro-Actionscited by 30
  21. 2019Composing Task-Agnostic Policies with Deep Reinforcement Learningcited by 20
  22. 2018Learning Representations in Model-Free Hierarchical Reinforcement Learningcited by 13
  23. 2020Language-Conditioned Goal Generation: a New Approach to Language Grounding for RLcited by 12
  24. 2018Computational Theories of Curiosity-Driven Learningcited by 11
  25. 2020Curious Hierarchical Actor-Critic Reinforcement Learningcited by 8
  26. 2019Compositional Transfer in Hierarchical Reinforcement Learningcited by 7
  27. 2018Hierarchical Reinforcement Learning with Abductive Planningcited by 5
  28. 2019Model Primitive Hierarchical Lifelong Reinforcement Learningcited by 5
  29. 2019From semantics to execution: Integrating action planning with reinforcement learning for robotic causal problem-solvingcited by 5
  30. 2020Improving Robot Dual-System Motor Learning with Intrinsically Motivated Meta-Control and Latent-Space Experience Imaginationcited by 5
  31. 2020Intrinsic Motivation and Episodic Memories for Robot Exploration of High-Dimensional Sensory Spacescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 9; 2025: 4; 2026: 2.

21 papers.

Sort:

By area:

  1. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
  2. 2025Skill Expansion and Composition in Parameter Spacecited by 14
  3. 2024PUZZLES: A Benchmark for Neural Algorithmic Reasoningcited by 13
  4. 2023Hierarchical Reinforcement Learning for Power Network Topology Controlcited by 11
  5. 2022Planning Irregular Object Packing via Hierarchical Reinforcement Learningcited by 8
  6. 2024Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planningcited by 7
  7. 2024Selective Exploration and Information Gathering in Search and Rescue Using Hierarchical Learning Guided by Natural Language Inputcited by 6
  8. 2025From Five Dimensions to Many: Large Language Models as Precise and Interpretable Psychological Profilerscited by 5
  9. 2023Habits and goals in synergy: a variational Bayesian framework for behaviorcited by 4
  10. 2024Episodic Reinforcement Learning with Expanded State-reward Spacecited by 3
  11. 2025Refining Hybrid Genetic Search for CVRP via Reinforcement Learning-Finetuned LLMcited by 3
  12. 2023Rejecting Cognitivism: Computational Phenomenology for Deep Learningcited by 2
9 more
  1. 2024Hierarchical Reinforcement Learning for Swarm Confrontation with High Uncertaintycited by 2
  2. 2022Language-Conditioned Reinforcement Learning to Solve Misunderstandings with Action Correctionscited by 1
  3. 2024Hierarchical Structure Enhances the Convergence and Generalizability of Linear Molecular Representationcited by 1
  4. 2024AgentForge: A Flexible Low-Code Platform for Reinforcement Learning Agent Designcited by 1
  5. 2024Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policiescited by 0
  6. 2024Scalable Hierarchical Reinforcement Learning for Hyper Scale Multi-Robot Task Planningcited by 0
  7. 2025Bidirectional Task-Motion Planning Based on Hierarchical Reinforcement Learning for Strategic Confrontationcited by 0
  8. 2026Performance-Driven Environment Abstraction with Multi-Timescale Learningcited by 0
  9. 2026Hierarchical Decision Making with Structured Policies: A Principled Design via Inverse Optimizationcited by 0