mlsanityData to 14 Oct 2026

Causality-driven Hierarchical Structure Discovery for Reinforcement Learning

arXiv 2210.06964 ↗First posted 13 Oct 2022cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Hierarchical reinforcement learning (HRL) effectively improves agents' exploration efficiency on tasks with sparse reward, with the guide of high-quality hierarchical structures (e.g., subgoals or options). However, how to automatically discover high-quality hierarchical structures is still a great challenge. Previous HRL methods can hardly discover the hierarchical structures in complex environments due to the low exploration efficiency by exploiting the randomness-driven exploration paradigm. To address this issue, we propose CDHRL, a causality-driven hierarchical reinforcement learning framework, leveraging a causality-driven discovery instead of a randomness-driven exploration to effectively build high-quality hierarchical structures in complicated environments. The key insight is that the causalities among environment variables are naturally fit for modeling reachable subgoals and their dependencies and can perfectly guide to build high-quality hierarchical structures. The results in two complex environments, 2D-Minecraft and Eden, show that CDHRL significantly boosts exploration efficiency with the causality-driven paradigm.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 26 of its 48 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

26 of 48 references.

References · 26

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2017Hindsight Experience Replaycited by 1,136
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  4. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  5. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  6. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  7. 2019Gradient-Based Neural DAG Learningcited by 227
  8. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  9. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  10. 2019Unsupervised State Representation Learning in Ataricited by 144
  11. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  12. 2019Learning Neural Causal Models from Unknown Interventionscited by 113
14 more
  1. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
  2. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102
  3. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  4. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  5. 2017Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learningcited by 83
  6. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 71
  7. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 68
  8. 2020Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learningcited by 53
  9. 2018Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependenciescited by 43
  10. 2020Align-RUDDER: Learning From Few Demonstrations by Reward Redistributioncited by 36
  11. 2020Disentangling causal effects for hierarchical reinforcement learningcited by 9
  12. 2019Hypothesis-Driven Skill Discovery for Hierarchical Deep Reinforcement Learningcited by 8
  13. 2021Hindsight Value Function for Variance Reduction in Stochastic Dynamic Environmentcited by 7
  14. 2021Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithmscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 6; 2025: 9; 2026: 3.

24 papers.

Sort:

By area:

  1. 2023A Survey on Causal Reinforcement Learningcited by 45
  2. 2023Automated Cyber Defence: A Reviewcited by 21
  3. 2023Context Shift Reduction for Offline Meta-Reinforcement Learningcited by 18
  4. 2024ADAM: An Embodied Causal Agent in Open-World Environmentscited by 13
  5. 2025Learning Interactive World Model for Object-Centric Reinforcement Learningcited by 11
  6. 2023Contrastive Modules with Temporal Attention for Multi-Task Reinforcement Learningcited by 10
  7. 2024Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learningcited by 9
  8. 2024SkiLD: Unsupervised Skill Discovery Guided by Factor Interactionscited by 9
  9. 2023Conceptual Reinforcement Learning for Language-Conditioned Taskscited by 8
  10. 2025Causally Aligned Curriculum Learningcited by 6
  11. 2024Variable-Agnostic Causal Exploration for Reinforcement Learningcited by 4
  12. 2024Learning Causal Dynamics Models in Object-Oriented Environmentscited by 3
12 more
  1. 2025CausalPlan: Empowering Efficient LLM Multi-Agent Collaboration Through Causality-Driven Planningcited by 2
  2. 2024Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledgecited by 1
  3. 2025D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detectioncited by 1
  4. 2023Causality is all you needcited by 0
  5. 2025Causality-Based Reinforcement Learning Method for Multi-Stage Robotic Taskscited by 0
  6. 2025Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challengescited by 0
  7. 2025Hierarchical Reinforcement Learning with Targeted Causal Interventionscited by 0
  8. 2025Goal Discovery with Causal Capacity for Efficient Reinforcement Learningcited by 0
  9. 2025Focused Skill Discovery: Learning to Control Specific State Variables while Minimizing Side Effectscited by 0
  10. 2026CausalGDP: Causality-Guided Diffusion Policies for Reinforcement Learningcited by 0
  11. 2026Delay-Empowered Causal Hierarchical Reinforcement Learningcited by 0
  12. 2026Agentic multi-fidelity learning of quasiparticle and excitonic propertiescited by 0