# Disentangling causal effects for hierarchical reinforcement learning

arXiv: https://arxiv.org/abs/2010.01351 · HTML: https://mlsanity.com/paper/2010.01351/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2020.

## Built on by (most cited first, 9 of 9)

- [D'ya like DAGs? A Survey on Structure Learning and Causal Discovery](https://mlsanity.com/paper/2103.02582/) (2021)
- [Causality-driven Hierarchical Structure Discovery for Reinforcement Learning](https://mlsanity.com/paper/2210.06964/) (2022)
- [Direct Advantage Estimation](https://mlsanity.com/paper/2109.06093/) (2021)
- [Skill or Luck? Return Decomposition via Advantage Functions](https://mlsanity.com/paper/2402.12874/) (2024)
- [Variable-Agnostic Causal Exploration for Reinforcement Learning](https://mlsanity.com/paper/2407.12437/) (2024)
- [Did I do that? Blame as a means to identify controlled effects in reinforcement learning](https://mlsanity.com/paper/2106.00266/) (2021)
- [CausalPlan: Empowering Efficient LLM Multi-Agent Collaboration Through Causality-Driven Planning](https://mlsanity.com/paper/2508.13721/) (2025)
- [Reducing Action Space for Deep Reinforcement Learning via Causal Effect Estimation](https://mlsanity.com/paper/2501.14543/) (2025)
- [Hierarchical Reinforcement Learning with Targeted Causal Interventions](https://mlsanity.com/paper/2507.04373/) (2025)

## Abstract (arXiv)

Exploration and credit assignment under sparse rewards are still challenging problems. We argue that these challenges arise in part due to the intrinsic rigidity of operating at the level of actions. Actions can precisely define how to perform an activity but are ill-suited to describe what activity to perform. Instead, causal effects are inherently composable and temporally abstract, making them ideal for descriptive tasks. By leveraging a hierarchy of causal effects, this study aims to expedite the learning of task-specific behavior and aid exploration. Borrowing counterfactual and normality measures from causal literature, we disentangle controllable effects from effects caused by other dynamics of the environment. We propose CEHRL, a hierarchical method that models the distribution of controllable effects using a Variational Autoencoder. This distribution is used by a high-level policy to 1) explore the environment via random effect exploration so that novel effects are continuously discovered and learned, and to 2) learn task-specific behavior by prioritizing the effects that maximize a given reward function. In comparison to exploring with random actions, experimental results show that random effect exploration is a more efficient mechanism and that by assigning credit to few effects rather than many actions, CEHRL learns tasks more rapidly.