# Causally Correct Partial Models for Reinforcement Learning

arXiv: https://arxiv.org/abs/2002.02836 · HTML: https://mlsanity.com/paper/2002.02836/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2020.

## Built on by (most cited first, 24 of 24)

- [D'ya like DAGs? A Survey on Structure Learning and Causal Discovery](https://mlsanity.com/paper/2103.02582/) (2021)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [Causal Influence Detection for Improving Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2106.03443/) (2021)
- [Shaking the foundations: delusions in sequence models for interaction and control](https://mlsanity.com/paper/2110.10819/) (2021)
- [Muesli: Combining Improvements in Policy Optimization](https://mlsanity.com/paper/2104.06159/) (2021)
- [Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning](https://mlsanity.com/paper/2107.00848/) (2021)
- [Vector Quantized Models for Planning](https://mlsanity.com/paper/2106.04615/) (2021)
- [A Survey on Causal Reinforcement Learning](https://mlsanity.com/paper/2302.05209/) (2023)
- [Tell me why! Explanations support learning relational and causal structure](https://mlsanity.com/paper/2112.03753/) (2021)
- [Procedural Generalization by Planning with Self-Supervised World Models](https://mlsanity.com/paper/2111.01587/) (2021)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [Automated Cyber Defence: A Review](https://mlsanity.com/paper/2303.04926/) (2023)
- [Causal Action Influence Aware Counterfactual Data Augmentation](https://mlsanity.com/paper/2405.18917/) (2024)
- [Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learning](https://mlsanity.com/paper/2406.03234/) (2024)
- [Do Transformer World Models Give Better Policy Gradients?](https://mlsanity.com/paper/2402.05290/) (2024)
- [On the Generalization and Adaption Performance of Causal Models](https://mlsanity.com/paper/2206.04620/) (2022)
- [Self-Consistent Models and Values](https://mlsanity.com/paper/2110.12840/) (2021)
- [Learning Causal Overhypotheses through Exploration in Children and Computational Models](https://mlsanity.com/paper/2202.10430/) (2022)
- [Deconfounding Imitation Learning with Variational Inference](https://mlsanity.com/paper/2211.02667/) (2022)
- [A Validation Tool for Designing Reinforcement Learning Environments](https://mlsanity.com/paper/2112.05519/) (2021)
- [REVEAL-IT: REinforcement learning with Visibility of Evolving Agent poLicy for InTerpretability](https://mlsanity.com/paper/2406.14214/) (2024)
- [Disentangled Representations for Causal Cognition](https://mlsanity.com/paper/2407.00744/) (2024)
- [CAIMAN: Causal Action Influence Detection for Sample-efficient Loco-manipulation](https://mlsanity.com/paper/2502.00835/) (2025)
- [Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challenges](https://mlsanity.com/paper/2505.08453/) (2025)

## Abstract (arXiv)

In reinforcement learning, we can learn a model of future observations and rewards, and use it to plan the agent's next actions. However, jointly modeling future observations can be computationally expensive or even intractable if the observations are high-dimensional (e.g. images). For this reason, previous works have considered partial models, which model only part of the observation. In this paper, we show that partial models can be causally incorrect: they are confounded by the observations they don't model, and can therefore lead to incorrect planning. To address this, we introduce a general family of partial models that are provably causally correct, yet remain fast because they do not need to fully model future observations.