# Causal Reasoning from Meta-reinforcement Learning

arXiv: https://arxiv.org/abs/1901.08162 · HTML: https://mlsanity.com/paper/1901.08162/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2019.

## Built on by (most cited first, 30 of 53)

- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning](https://mlsanity.com/paper/2107.00848/) (2021)
- [A Survey on Causal Reinforcement Learning](https://mlsanity.com/paper/2302.05209/) (2023)
- [Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning](https://mlsanity.com/paper/2107.12544/) (2021)
- [Visual cognition in multimodal large language models](https://mlsanity.com/paper/2311.16093/) (2023)
- [Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents](https://mlsanity.com/paper/2102.02926/) (2021)
- [Tell me why! Explanations support learning relational and causal structure](https://mlsanity.com/paper/2112.03753/) (2021)
- [The broader spectrum of in-context learning](https://mlsanity.com/paper/2412.03782/) (2024)
- [Counterfactual harm](https://mlsanity.com/paper/2204.12993/) (2022)
- [Deductive Beam Search: Decoding Deducible Rationale for Chain-of-Thought Reasoning](https://mlsanity.com/paper/2401.17686/) (2024)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [Interventional Sum-Product Networks: Causal Inference with Tractable Probabilistic Models](https://mlsanity.com/paper/2102.10440/) (2021)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [Passive learning of active causal strategies in agents and language models](https://mlsanity.com/paper/2305.16183/) (2023)
- [Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machines](https://mlsanity.com/paper/2205.11558/) (2022)
- [Causal Deep Learning](https://mlsanity.com/paper/2303.02186/) (2023)
- [Causal models in string diagrams](https://mlsanity.com/paper/2304.07638/) (2023)
- [PUZZLES: A Benchmark for Neural Algorithmic Reasoning](https://mlsanity.com/paper/2407.00401/) (2024)
- [Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face](https://mlsanity.com/paper/2508.06811/) (2025)
- [A Meta-Reinforcement Learning Algorithm for Causal Discovery](https://mlsanity.com/paper/2207.08457/) (2022)
- [Beyond Object Recognition: A New Benchmark towards Object Concept Learning](https://mlsanity.com/paper/2212.02710/) (2022)
- [Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulation](https://mlsanity.com/paper/2402.07127/) (2024)
- [Debiased Multimodal Understanding for Human Language Sequences](https://mlsanity.com/paper/2403.05025/) (2024)
- [On the Generalization and Adaption Performance of Causal Models](https://mlsanity.com/paper/2206.04620/) (2022)
- [Learning Causal Overhypotheses through Exploration in Children and Computational Models](https://mlsanity.com/paper/2202.10430/) (2022)
- [Disentangling Abstraction from Statistical Pattern Matching in Human and Machine Learning](https://mlsanity.com/paper/2204.01437/) (2022)
- [Intelligent Computing: The Latest Advances, Challenges and Future](https://mlsanity.com/paper/2211.11281/) (2022)
- [A Survey of Methods, Challenges and Perspectives in Causality](https://mlsanity.com/paper/2302.00293/) (2023)
- [Causal Counterfactuals for Improving the Robustness of Reinforcement Learning](https://mlsanity.com/paper/2211.05551/) (2022)

## Abstract (arXiv)

Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether causal reasoning can emerge via meta-reinforcement learning. We train a recurrent network with model-free reinforcement learning to solve a range of problems that each contain causal structure. We find that the trained agent can perform causal reasoning in novel situations in order to obtain rewards. The agent can select informative interventions, draw causal inferences from observational data, and make counterfactual predictions. Although established formal causal reasoning algorithms also exist, in this paper we show that such reasoning can arise from model-free reinforcement learning, and suggest that causal reasoning in complex settings may benefit from the more end-to-end learning-based approaches presented here. This work also offers new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform -- and interpret -- experiments.