# Causal Counterfactuals for Improving the Robustness of Reinforcement Learning

arXiv: https://arxiv.org/abs/2211.05551 · HTML: https://mlsanity.com/paper/2211.05551/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 58 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [European Union regulations on algorithmic decision-making and a "right to explanation"](https://mlsanity.com/paper/1606.08813/) (2016)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [Programmatically Interpretable Reinforcement Learning](https://mlsanity.com/paper/1804.02477/) (2018)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [Causal Discovery with Reinforcement Learning](https://mlsanity.com/paper/1906.04477/) (2019)
- [Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping](https://mlsanity.com/paper/1709.07857/) (2017)
- [Explainable Reinforcement Learning: A Survey](https://mlsanity.com/paper/2005.06247/) (2020)
- [Learning Neural Causal Models from Unknown Interventions](https://mlsanity.com/paper/1910.01075/) (2019)
- [CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning](https://mlsanity.com/paper/2010.04296/) (2020)
- [Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey](https://mlsanity.com/paper/2005.00935/) (2020)
- [Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1312.6082/) (2013)
- [CoPhy: Counterfactual Learning of Physical Dynamics](https://mlsanity.com/paper/1909.12000/) (2019)
- [Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning](https://mlsanity.com/paper/1712.07294/) (2017)
- [Neural Production Systems: Learning Rule-Governed Visual Dynamics](https://mlsanity.com/paper/2103.01937/) (2021)
- [Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees](https://mlsanity.com/paper/1807.05887/) (2018)
- [Causal Reasoning from Meta-reinforcement Learning](https://mlsanity.com/paper/1901.08162/) (2019)
- [Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning](https://mlsanity.com/paper/2010.03110/) (2020)
- [Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey](https://mlsanity.com/paper/2108.09003/) (2021)
- [Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario](https://mlsanity.com/paper/2006.13615/) (2020)
- [Sim-to-Real Transfer with Incremental Environment Complexity for Reinforcement Learning of Depth-Based Robot Navigation](https://mlsanity.com/paper/2004.14684/) (2020)

## Built on by (most cited first, 5 of 5)

- [Causal Deep Learning](https://mlsanity.com/paper/2303.02186/) (2023)
- [Causality from Bottom to Top: A Survey](https://mlsanity.com/paper/2403.11219/) (2024)
- [Exploring CausalWorld: Enhancing robotic manipulation via knowledge transfer and curriculum learning](https://mlsanity.com/paper/2403.17266/) (2024)
- [Self-Evolving Cognitive Framework via Causal World Modeling for Embodied Scientific Intelligence](https://mlsanity.com/paper/2606.22449/) (2026)
- [Generating Causal Explanations of Vehicular Agent Behavioural Interactions with Learnt Reward Profiles](https://mlsanity.com/paper/2503.14557/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) is used in various robotic applications. RL enables agents to learn tasks autonomously by interacting with the environment. The more critical the tasks are, the higher the demand for the robustness of the RL systems. Causal RL combines RL and causal inference to make RL more robust. Causal RL agents use a causal representation to capture the invariant causal mechanisms that can be transferred from one task to another. Currently, there is limited research in Causal RL, and existing solutions are usually not complete or feasible for real-world applications. In this work, we propose CausalCF, the first complete Causal RL solution incorporating ideas from Causal Curiosity and CoPhy. Causal Curiosity provides an approach for using interventions, and CoPhy is modified to enable the RL agent to perform counterfactuals. Causal Curiosity has been applied to robotic grasping and manipulation tasks in CausalWorld. CausalWorld provides a realistic simulation environment based on the TriFinger robot. We apply CausalCF to complex robotic tasks and show that it improves the RL agent's robustness using CausalWorld.