mlsanityData to 7 Oct 2026

Causal Influence Detection for Improving Efficiency in Reinforcement Learning

arXiv 2106.03443 ↗First posted 7 Jun 2021cs.LGcited by 70

Not on the map. Why?

Abstract (arXiv)

Many reinforcement learning (RL) environments consist of independent entities that interact sparsely. In such environments, RL agents have only limited influence over other entities in any particular situation. Our idea in this work is that learning can be efficiently guided by knowing when and what the agent can influence with its actions. To achieve this, we introduce a measure of situation-dependent causal influence based on conditional mutual information and show that it can reliably detect states of influence. We then propose several ways to integrate this measure into RL algorithms to improve exploration and off-policy learning. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 38 of its 70 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

38 of 70 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  10. 2018Learning Dexterous In-Hand Manipulationcited by 643
  11. 2019On Variational Bounds of Mutual Informationcited by 550
  12. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 408
26 more
  1. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 273
  2. 2019Self-Supervised Exploration via Disagreementcited by 263
  3. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  4. 2020Weakly-Supervised Disentanglement Without Compromisescited by 237
  5. 2019Recurrent Independent Mechanismscited by 219
  6. 2021Towards Causal Representation Learningcited by 207
  7. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  8. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  9. 2019Learning Neural Causal Models from Unknown Interventionscited by 113
  10. 2017Learning Independent Causal Mechanismscited by 104
  11. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  12. 2018Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Searchcited by 81
  13. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 78
  14. 2017Conditional independence testing based on a nearest-neighbor estimator of conditional mutual informationcited by 70
  15. 2013Empowerment -- an Introductioncited by 65
  16. 2019Exploration via Hindsight Goal Generationcited by 53
  17. 2020Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learningcited by 53
  18. 2014Efficient Estimation of Mutual Information for Strongly Dependent Variablescited by 46
  19. 2018Deconfounding Reinforcement Learning in Observational Settingscited by 36
  20. 2018Contingency-Aware Exploration in Reinforcement Learningcited by 26
  21. 2020Causally Correct Partial Models for Reinforcement Learningcited by 24
  22. 2018Energy-Based Hindsight Experience Prioritizationcited by 20
  23. 2021Mutual Information State Intrinsic Controlcited by 18
  24. 2019Mega-Reward: Achieving Human-Level Play without Extrinsic Rewardscited by 5
  25. 2013Information driven self-organization of complex robotic behaviorscited by 4
  26. 2013Linear combination of one-step predictive information with an external reward in an episodic policy gradient setting: a critical analysiscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 11; 2024: 18; 2025: 23; 2026: 6.

70 papers.

Sort:

By area:

  1. 2022On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networkscited by 119
  2. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 64
  3. 2022Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systemscited by 50
  4. 2023A Survey on Causal Reinforcement Learningcited by 43
  5. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33
  6. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  7. 2024Unifying Causal Representation Learning with the Invariance Principlecited by 30
  8. 2022Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulationcited by 29
  9. 2026Causal-JEPA: Learning World Models through Object-Level Latent Maskingcited by 25
  10. 2022Causality-driven Hierarchical Structure Discovery for Reinforcement Learningcited by 23
  11. 2023Explainable Reinforcement Learning via a Causal World Modelcited by 22
  12. 2023Causal Reinforcement Learning: A Surveycited by 22
38 more
  1. 2023Causal Deep Learningcited by 20
  2. 2021Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domainscited by 18
  3. 2021Self-supervised Reinforcement Learning with Independently Controllable Subgoalscited by 16
  4. 2022Variational Causal Dynamics: Discovering Modular World Models from Interventionscited by 16
  5. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  6. 2022Causal Discovery of Dynamic Models for Predicting Human Spatial Interactionscited by 13
  7. 2024Causal Action Influence Aware Counterfactual Data Augmentationcited by 12
  8. 2024Entity-based Reinforcement Learning for Autonomous Cyber Defencecited by 11
  9. 2025Learning Interactive World Model for Object-Centric Reinforcement Learningcited by 11
  10. 2023Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learningcited by 10
  11. 2024Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learningcited by 10
  12. 2024SPARTAN: A Sparse Transformer World Model Attending to What Matterscited by 10
  13. 2024SkiLD: Unsupervised Skill Discovery Guided by Factor Interactionscited by 9
  14. 2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
  15. 2023ELDEN: Exploration via Local Dependenciescited by 8
  16. 2023Granger Causal Interaction Skill Chainscited by 5
  17. 2024CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systemscited by 5
  18. 2025From Curiosity to Competence: How World Models Interact with the Dynamics of Explorationcited by 5
  19. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  20. 2022Estimating Regression Predictive Distributions with Sample Networkscited by 4
  21. 2023A Neuro-Symbolic Approach for Enhanced Human Motion Predictioncited by 4
  22. 2024On Discovery of Local Independence over Continuous Variables via Neural Contextual Decompositioncited by 4
  23. 2024Variable-Agnostic Causal Exploration for Reinforcement Learningcited by 4
  24. 2025Towards Empowerment Gain through Causal Structure Learning in Model-Based RLcited by 4
  25. 2025Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learningcited by 4
  26. 2025Curious Causality-Seeking Agents Learn Meta Causal Worldcited by 4
  27. 2024Automated Discovery of Functional Actual Causes in Complex Environmentscited by 3
  28. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3
  29. 2026Disentangling Dynamical Systems: Causal Representation Learning Meets Local Sparse Attentioncited by 3
  30. 2023Q-Cogni: An Integrated Causal Reinforcement Learning Frameworkcited by 2
  31. 2025Behaviour Discovery and Attribution for Explainable Reinforcement Learningcited by 2
  32. 2025Causality-informed Anomaly Detection in Partially Observable Sensor Networks: Moving beyond Correlationscited by 2
  33. 2025CausalPlan: Empowering Efficient LLM Multi-Agent Collaboration Through Causality-Driven Planningcited by 2
  34. 2025From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learningcited by 2
  35. 2022Using Contrastive Samples for Identifying and Leveraging Possible Causal Relationships in Reinforcement Learningcited by 1
  36. 2024Reward-Relevance-Filtered Linear Offline Reinforcement Learningcited by 1
  37. 2024Fundamental Properties of Causal Entropy and Information Gaincited by 1
  38. 2024Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledgecited by 1

50 most cited of 70 All 70 (JSON)