# Causal Influence Detection for Improving Efficiency in Reinforcement Learning

arXiv: https://arxiv.org/abs/2106.03443 · HTML: https://mlsanity.com/paper/2106.03443/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 70 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [Self-Supervised Exploration via Disagreement](https://mlsanity.com/paper/1906.04161/) (2019)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [Recurrent Independent Mechanisms](https://mlsanity.com/paper/1909.10893/) (2019)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms](https://mlsanity.com/paper/1901.10912/) (2019)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [Learning Neural Causal Models from Unknown Interventions](https://mlsanity.com/paper/1910.01075/) (2019)
- [Learning Independent Causal Mechanisms](https://mlsanity.com/paper/1712.00961/) (2017)
- [Counterfactual Data Augmentation using Locally Factored Dynamics](https://mlsanity.com/paper/2007.02863/) (2020)
- [Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search](https://mlsanity.com/paper/1811.06272/) (2018)
- [CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/1810.06284/) (2018)
- [Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information](https://mlsanity.com/paper/1709.01447/) (2017)
- [Empowerment -- an Introduction](https://mlsanity.com/paper/1310.1863/) (2013)
- [Exploration via Hindsight Goal Generation](https://mlsanity.com/paper/1906.04279/) (2019)
- [Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning](https://mlsanity.com/paper/2010.03110/) (2020)
- [Efficient Estimation of Mutual Information for Strongly Dependent Variables](https://mlsanity.com/paper/1411.2003/) (2014)

## Built on by (most cited first, 30 of 71)

- [On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks](https://mlsanity.com/paper/2203.09168/) (2022)
- [Causal Dynamics Learning for Task-Independent State Abstraction](https://mlsanity.com/paper/2206.13452/) (2022)
- [Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systems](https://mlsanity.com/paper/2206.06169/) (2022)
- [A Survey on Causal Reinforcement Learning](https://mlsanity.com/paper/2302.05209/) (2023)
- [Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning](https://mlsanity.com/paper/2207.09081/) (2022)
- [The Essential Role of Causality in Foundation World Models for Embodied AI](https://mlsanity.com/paper/2402.06665/) (2024)
- [Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulation](https://mlsanity.com/paper/2206.11403/) (2022)
- [Unifying Causal Representation Learning with the Invariance Principle](https://mlsanity.com/paper/2409.02772/) (2024)
- [Causal-JEPA: Learning World Models through Object-Level Latent Masking](https://mlsanity.com/paper/2602.11389/) (2026)
- [Causality-driven Hierarchical Structure Discovery for Reinforcement Learning](https://mlsanity.com/paper/2210.06964/) (2022)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [Explainable Reinforcement Learning via a Causal World Model](https://mlsanity.com/paper/2305.02749/) (2023)
- [Causal Deep Learning](https://mlsanity.com/paper/2303.02186/) (2023)
- [Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domains](https://mlsanity.com/paper/2110.15949/) (2021)
- [Self-supervised Reinforcement Learning with Independently Controllable Subgoals](https://mlsanity.com/paper/2109.04150/) (2021)
- [Variational Causal Dynamics: Discovering Modular World Models from Interventions](https://mlsanity.com/paper/2206.11131/) (2022)
- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Causal Discovery of Dynamic Models for Predicting Human Spatial Interactions](https://mlsanity.com/paper/2210.16535/) (2022)
- [Causal Action Influence Aware Counterfactual Data Augmentation](https://mlsanity.com/paper/2405.18917/) (2024)
- [Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learning](https://mlsanity.com/paper/2406.03234/) (2024)
- [Entity-based Reinforcement Learning for Autonomous Cyber Defence](https://mlsanity.com/paper/2410.17647/) (2024)
- [Learning Interactive World Model for Object-Centric Reinforcement Learning](https://mlsanity.com/paper/2511.02225/) (2025)
- [Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2312.09539/) (2023)
- [SPARTAN: A Sparse Transformer World Model Attending to What Matters](https://mlsanity.com/paper/2411.06890/) (2024)
- [SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions](https://mlsanity.com/paper/2410.18416/) (2024)
- [Uncertainty-Aware Decision Transformer for Stochastic Driving Environments](https://mlsanity.com/paper/2309.16397/) (2023)
- [ELDEN: Exploration via Local Dependencies](https://mlsanity.com/paper/2310.08702/) (2023)
- [Granger Causal Interaction Skill Chains](https://mlsanity.com/paper/2306.09509/) (2023)
- [Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2505.03172/) (2025)
- [CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systems](https://mlsanity.com/paper/2406.17300/) (2024)

## Abstract (arXiv)

Many reinforcement learning (RL) environments consist of independent entities that interact sparsely. In such environments, RL agents have only limited influence over other entities in any particular situation. Our idea in this work is that learning can be efficiently guided by knowing when and what the agent can influence with its actions. To achieve this, we introduce a measure of situation-dependent causal influence based on conditional mutual information and show that it can reliably detect states of influence. We then propose several ways to integrate this measure into RL algorithms to improve exploration and off-policy learning. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.