mlsanityData to 14 Oct 2026

D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detection

arXiv 2505.01979 ↗First posted 4 May 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Current Hierarchical Reinforcement Learning (HRL) algorithms excel in long-horizon sequential decision-making tasks but still face two challenges: delay effects and spurious correlations. To address them, we propose a causal HRL approach called D3HRL. First, D3HRL models delayed effects as causal relationships across different time spans and employs distributed causal discovery to learn these relationships. Second, it employs conditional independence testing to eliminate spurious correlations. Finally, D3HRL constructs and trains hierarchical policies based on the identified true causal relationships. These three steps are iteratively executed, gradually exploring the complete causal chain of the task. Experiments conducted in 2D-MineCraft and MiniGrid show that D3HRL demonstrates superior sensitivity to delay effects and accurately identifies causal relationships, leading to reliable decision-making in complex environments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 52 references that are in the map. First posted on arXiv 4 May 2025.

Builds on

28 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Variable-Agnostic Causal Exploration for Reinforcement Learning17 Jul

Other references · 27

  1. 2017Hindsight Experience Replaycited by 1,136
  2. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  3. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  4. 2016The Option-Critic Architecturecited by 448
  5. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  6. 2024Reinforcement learningcited by 247
  7. 2017Detecting causal associations in large nonlinear time series datasetscited by 244
  8. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  9. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  10. 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 150
  11. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  12. 2018Variational Option Discovery Algorithmscited by 122
15 more
  1. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
  2. 2019Learning Neural Causal Models from Unknown Interventionscited by 113
  3. 2017Meta Learning Shared Hierarchiescited by 93
  4. 2018Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Searchcited by 82
  5. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 65
  6. 2018Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependenciescited by 43
  7. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 34
  8. 2022Causality-driven Hierarchical Structure Discovery for Reinforcement Learningcited by 24
  9. 2023Causal Reinforcement Learning: A Surveycited by 23
  10. 2020Efficient Reinforcement Learning in Factored MDPs with Application to Constrained RLcited by 16
  11. 2020Curious Hierarchical Actor-Critic Reinforcement Learningcited by 8
  12. 2023LESSON: Learning to Integrate Exploration Strategies for Reinforcement Learning via an Option Frameworkcited by 7
  13. 2020Reinforcement Learning of Causal Variables Using Mediation Analysiscited by 6
  14. 2023Granger Causal Interaction Skill Chainscited by 6
  15. 2018Diversity-Driven Extensible Hierarchical Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Delay-Empowered Causal Hierarchical Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Hierarchical Reinforcement Learning with Targeted Causal Interventions
  2. 2026CausalGDP: Causality-Guided Diffusion Policies for Reinforcement Learning
  3. 2025Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications
  4. 2025Learning Nonlinear Causal Reductions to Explain Reinforcement Learning Policies
  5. 2024Hierarchical Decision Making Based on Structural Information Principles
  6. 2025Causal-Paced Deep Reinforcement Learning
  7. 2024Learning by Doing: An Online Causal Reinforcement Learning Framework with Causal-Aware Policy
  8. 2025A Roadmap Towards Improving Multi-Agent Reinforcement Learning With Causal Discovery And Inference