mlsanityData to 14 Oct 2026

Hierarchical Reinforcement Learning with Targeted Causal Interventions

arXiv 2507.04373 ↗First posted 6 Jul 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Hierarchical reinforcement learning (HRL) improves the efficiency of long-horizon reinforcement-learning tasks with sparse rewards by decomposing the task into a hierarchy of subgoals. The main challenge of HRL is efficient discovery of the hierarchical structure among subgoals and utilizing this structure to achieve the final goal. We address this challenge by modeling the subgoal structure as a causal graph and propose a causal discovery algorithm to learn it. Additionally, rather than intervening on the subgoals at random during exploration, we harness the discovered causal model to prioritize subgoal interventions based on their importance in attaining the final goal. These targeted interventions result in a significantly more efficient policy in terms of the training cost. Unlike previous work on causal HRL, which lacked theoretical analysis, we provide a formal analysis of the problem. Specifically, for tree structures and, for a variant of ErdHos-R'enyi random graphs, our approach results in remarkable improvements. Our experimental results on HRL tasks also illustrate that our proposed framework outperforms existing work in terms of training cost.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 31 of its 50 references that are in the map. First posted on arXiv 6 Jul 2025.

Builds on

31 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Causally Aligned Curriculum Learning21 Mar
  2. 2024Variable-Agnostic Causal Exploration for Reinforcement Learning17 Jul

Other references · 29

  1. 2017Hindsight Experience Replaycited by 1,136
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  3. 2016The Option-Critic Architecturecited by 448
  4. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  5. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  6. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  7. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  8. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  9. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  10. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  11. 2019Learning Neural Causal Models from Unknown Interventionscited by 113
  12. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
17 more
  1. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  2. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  3. 2017Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learningcited by 83
  4. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 71
  5. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 65
  6. 2016Learning and Transfer of Modulated Locomotor Controllerscited by 59
  7. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  8. 2019Neural Logic Reinforcement Learningcited by 53
  9. 2018Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependenciescited by 43
  10. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 34
  11. 2021Neuro-Symbolic Reinforcement Learning with First-Order Logiccited by 30
  12. 2022Causality-driven Hierarchical Structure Discovery for Reinforcement Learningcited by 24
  13. 2023Explainable Reinforcement Learning via a Causal World Modelcited by 22
  14. 2022Offline Reinforcement Learning with Causal Structured World Modelscited by 19
  15. 2018Policy Search in Continuous Action Domains: an Overviewcited by 10
  16. 2020Disentangling causal effects for hierarchical reinforcement learningcited by 9
  17. 2024Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledgecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detection
  2. 2026CausalGDP: Causality-Guided Diffusion Policies for Reinforcement Learning
  3. 2026On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning
  4. 2025Learning Nonlinear Causal Reductions to Explain Reinforcement Learning Policies
  5. 2024Learning by Doing: An Online Causal Reinforcement Learning Framework with Causal-Aware Policy
  6. 2024CORE: Towards Scalable and Efficient Causal Discovery with Reinforcement Learning
  7. 2024MENTOR: Guiding Hierarchical Reinforcement Learning with Human Feedback and Dynamic Distance Constraint
  8. 2025Causal-Paced Deep Reinforcement Learning