# Towards Empowerment Gain through Causal Structure Learning in Model-Based RL

arXiv: https://arxiv.org/abs/2502.10077 · HTML: https://mlsanity.com/paper/2502.10077/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 38 of its 51 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 38)

- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [Causal Discovery from Heterogeneous/Nonstationary Data with Independent Changes](https://mlsanity.com/paper/1903.01672/) (2019)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [Robust agents learn causal world models](https://mlsanity.com/paper/2402.10877/) (2024)
- [Causal Influence Detection for Improving Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2106.03443/) (2021)
- [Empowerment -- an Introduction](https://mlsanity.com/paper/1310.1863/) (2013)
- [Denoised MDPs: Learning World Models Better Than the World Itself](https://mlsanity.com/paper/2206.15477/) (2022)
- [Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction](https://mlsanity.com/paper/2007.14535/) (2020)
- [Causal Dynamics Learning for Task-Independent State Abstraction](https://mlsanity.com/paper/2206.13452/) (2022)
- [Temporal Predictive Coding For Model-Based Planning In Latent Space](https://mlsanity.com/paper/2106.07156/) (2021)
- [Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning](https://mlsanity.com/paper/2107.00848/) (2021)
- [AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning](https://mlsanity.com/paper/2107.02729/) (2021)
- [On the model-based stochastic value gradient for continuous reinforcement learning](https://mlsanity.com/paper/2008.12775/) (2020)
- [MoCoDA: Model-based Counterfactual Data Augmentation](https://mlsanity.com/paper/2210.11287/) (2022)
- [Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning](https://mlsanity.com/paper/2207.09081/) (2022)
- [Action-Sufficient State Representation Learning for Control with Structural Constraints](https://mlsanity.com/paper/2110.05721/) (2021)
- [A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2106.02097/) (2021)
- [INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL](https://mlsanity.com/paper/2204.08585/) (2022)
- [Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation](https://mlsanity.com/paper/2307.07907/) (2023)
- [Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Critic](https://mlsanity.com/paper/2112.10504/) (2021)
- [Learning World Models with Identifiable Factorization](https://mlsanity.com/paper/2306.06561/) (2023)
- [Composing Pre-Trained Object-Centric Representations for Robotics From "What" and "Where" Foundation Models](https://mlsanity.com/paper/2404.13474/) (2024)
- [A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment](https://mlsanity.com/paper/1907.12392/) (2019)

## Built on by (most cited first, 5 of 5)

- [Learning Interactive World Model for Object-Centric Reinforcement Learning](https://mlsanity.com/paper/2511.02225/) (2025)
- [Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling](https://mlsanity.com/paper/2607.04409/) (2026)
- [Ada-Diffuser: Latent-Aware Adaptive Diffusion for Decision-Making](https://mlsanity.com/paper/2605.16054/) (2026)
- [Delay-Empowered Causal Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2605.12261/) (2026)
- [Transferable Delay-Aware Reinforcement Learning via Implicit Causal Graph Modeling](https://mlsanity.com/paper/2605.12312/) (2026)

## Abstract (arXiv)

In Model-Based Reinforcement Learning (MBRL), incorporating causal structures into dynamics models provides agents with a structured understanding of the environments, enabling efficient decision. Empowerment as an intrinsic motivation enhances the ability of agents to actively control their environments by maximizing the mutual information between future states and actions. We posit that empowerment coupled with causal understanding can improve controllability, while enhanced empowerment gain can further facilitate causal reasoning in MBRL. To improve learning efficiency and controllability, we propose a novel framework, Empowerment through Causal Learning (ECL), where an agent with the awareness of causal dynamics models achieves empowerment-driven exploration and optimizes its causal structure for task learning. Specifically, ECL operates by first training a causal dynamics model of the environment based on collected data. We then maximize empowerment under the causal structure for exploration, simultaneously using data gathered through exploration to update causal dynamics model to be more controllable than dense dynamics model without causal structure. In downstream task learning, an intrinsic curiosity reward is included to balance the causality, mitigating overfitting. Importantly, ECL is method-agnostic and is capable of integrating various causal discovery methods. We evaluate ECL combined with 3 causal discovery methods across 6 environments including pixel-based tasks, demonstrating its superior performance compared to other causal MBRL methods, in terms of causal discovery, sample efficiency, and asymptotic performance.