# Explainable Reinforcement Learning via a Causal World Model

arXiv: https://arxiv.org/abs/2305.02749 · HTML: https://mlsanity.com/paper/2305.02749/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 29 of its 56 references that are in the map.
- First posted on arXiv 4 May 2023.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 29 of 29)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [Explainable Reinforcement Learning: A Survey](https://mlsanity.com/paper/2005.06247/) (2020)
- [Invariant Causal Prediction for Block MDPs](https://mlsanity.com/paper/2003.06016/) (2020)
- [Causal Influence Detection for Improving Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2106.03443/) (2021)
- [Causal Dynamics Learning for Task-Independent State Abstraction](https://mlsanity.com/paper/2206.13452/) (2022)
- [Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning](https://mlsanity.com/paper/1912.05743/) (2019)
- [Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning](https://mlsanity.com/paper/2107.00848/) (2021)
- [Shapley Explanation Networks](https://mlsanity.com/paper/2104.02297/) (2021)
- [Provably Efficient Causal Reinforcement Learning with Confounded Observational Data](https://mlsanity.com/paper/2006.12311/) (2020)
- [Learning Finite State Representations of Recurrent Policy Networks](https://mlsanity.com/paper/1811.12530/) (2018)
- [Deconfounding Reinforcement Learning in Observational Settings](https://mlsanity.com/paper/1812.10576/) (2018)
- [Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning](https://mlsanity.com/paper/2207.09081/) (2022)
- [Causal Induction from Visual Observations for Goal Directed Tasks](https://mlsanity.com/paper/1910.01751/) (2019)
- [Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learning](https://mlsanity.com/paper/1806.01584/) (2018)
- [Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based robotics](https://mlsanity.com/paper/1901.08651/) (2019)
- [What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomes](https://mlsanity.com/paper/2011.05064/) (2020)
- [Fast Conditional Independence Test for Vector Variables with Large Sample Sizes](https://mlsanity.com/paper/1804.02747/) (2018)
- [Free-Lunch Saliency via Attention in Atari Agents](https://mlsanity.com/paper/1908.02511/) (2019)
- [Autonomous Self-Explanation of Behavior for Interactive Reinforcement Learning Agents](https://mlsanity.com/paper/1810.08811/) (2018)
- [Training a Resilient Q-Network against Observational Interference](https://mlsanity.com/paper/2102.09677/) (2021)

## Built on by (most cited first, 22 of 22)

- [Artificial General Intelligence (AGI)-Native Wireless Systems: A Journey Beyond 6G](https://mlsanity.com/paper/2405.02336/) (2024)
- [Making Large Language Models into World Models with Precondition and Effect Knowledge](https://mlsanity.com/paper/2409.12278/) (2024)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [Continual Learning, Not Training: Online Adaptation For Agents](https://mlsanity.com/paper/2511.01093/) (2025)
- [Learning Causal Dynamics Models in Object-Oriented Environments](https://mlsanity.com/paper/2405.12615/) (2024)
- [Learning Local Causal World Models with State Space Models and Attention](https://mlsanity.com/paper/2505.02074/) (2025)
- [ProPlay: Procedural World Models for Self-Evolving LLM Agents](https://mlsanity.com/paper/2606.12780/) (2026)
- [Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledge](https://mlsanity.com/paper/2407.20506/) (2024)
- [Towards Causal Model-Based Policy Optimization](https://mlsanity.com/paper/2503.09719/) (2025)
- [Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications](https://mlsanity.com/paper/2512.18135/) (2025)
- [Self-Evolving Cognitive Framework via Causal World Modeling for Embodied Scientific Intelligence](https://mlsanity.com/paper/2606.22449/) (2026)
- [Transformers and Slot Encoding for Sample Efficient Physical World Modelling](https://mlsanity.com/paper/2405.20180/) (2024)
- [Compositional Concept-Based Neuron-Level Interpretability for Deep Reinforcement Learning](https://mlsanity.com/paper/2502.00684/) (2025)
- [Generating Causal Explanations of Vehicular Agent Behavioural Interactions with Learnt Reward Profiles](https://mlsanity.com/paper/2503.14557/) (2025)
- [Hierarchical Reinforcement Learning with Targeted Causal Interventions](https://mlsanity.com/paper/2507.04373/) (2025)
- [Understanding Action Effects through Instrumental Empowerment in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2508.15652/) (2025)
- [Causal Model-Based Reinforcement Learning for Sample-Efficient IoT Channel Access](https://mlsanity.com/paper/2511.10291/) (2025)
- [CausalGDP: Causality-Guided Diffusion Policies for Reinforcement Learning](https://mlsanity.com/paper/2602.09207/) (2026)
- [Continual learning and refinement of causal models through dynamic predicate invention](https://mlsanity.com/paper/2602.17217/) (2026)
- [Transferable Delay-Aware Reinforcement Learning via Implicit Causal Graph Modeling](https://mlsanity.com/paper/2605.12312/) (2026)
- [Sample-Efficient Learning of Probabilistic Causes for Reachability in Markov Decision Processes with Probabilistic Guarantees](https://mlsanity.com/paper/2606.29681/) (2026)
- [When Do Causal World Models Help Modular LLM Agents](https://mlsanity.com/paper/2610.00012/) (2026)

## Abstract (arXiv)

Generating explanations for reinforcement learning (RL) is challenging as actions may produce long-term effects on the future. In this paper, we develop a novel framework for explainable RL by learning a causal world model without prior knowledge of the causal structure of the environment. The model captures the influence of actions, allowing us to interpret the long-term effects of actions through causal chains, which present how actions influence environmental variables and finally lead to rewards. Different from most explanatory models which suffer from low accuracy, our model remains accurate while improving explainability, making it applicable in model-based learning. As a result, we demonstrate that our causal model can serve as the bridge between explainability and learning.