# Causal Discovery with Reinforcement Learning

arXiv: https://arxiv.org/abs/1906.04477 · HTML: https://mlsanity.com/paper/1906.04477/

## Key facts

- Cited by 145 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 30 of 145)

- [D'ya like DAGs? A Survey on Structure Learning and Causal Discovery](https://mlsanity.com/paper/2103.02582/) (2021)
- [Bayesian Structure Learning with Generative Flow Networks](https://mlsanity.com/paper/2202.13903/) (2022)
- [DAGMA: Learning DAGs via M-matrices and a Log-Determinant Acyclicity Characterization](https://mlsanity.com/paper/2209.08037/) (2022)
- [Score matching enables causal discovery of nonlinear additive noise models](https://mlsanity.com/paper/2203.04413/) (2022)
- [Amortized Inference for Causal Structure Learning](https://mlsanity.com/paper/2205.12934/) (2022)
- [Efficient Neural Causal Discovery without Acyclicity Constraints](https://mlsanity.com/paper/2107.10483/) (2021)
- [Learning to Induce Causal Structure](https://mlsanity.com/paper/2204.04875/) (2022)
- [gCastle: A Python Toolbox for Causal Discovery](https://mlsanity.com/paper/2111.15155/) (2021)
- [Ordering-Based Causal Discovery with Reinforcement Learning](https://mlsanity.com/paper/2105.06631/) (2021)
- [DAGs with No Curl: An Efficient DAG Structure Learning Approach](https://mlsanity.com/paper/2106.07197/) (2021)
- [Out-of-distribution Generalization with Causal Invariant Transformations](https://mlsanity.com/paper/2203.11528/) (2022)
- [Causal Discovery with Score Matching on Additive Models with Arbitrary Noise](https://mlsanity.com/paper/2304.03265/) (2023)
- [Learning Neural Causal Models with Active Interventions](https://mlsanity.com/paper/2109.02429/) (2021)
- [A Survey on Causal Discovery Methods for I.I.D. and Time Series Data](https://mlsanity.com/paper/2303.15027/) (2023)
- [Reinforced Causal Explainer for Graph Neural Networks](https://mlsanity.com/paper/2204.11028/) (2022)
- [Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning](https://mlsanity.com/paper/2207.09081/) (2022)
- [Contrastive Video Question Answering via Video Graph Transformer](https://mlsanity.com/paper/2302.13668/) (2023)
- [Neurosymbolic Reinforcement Learning and Planning: A Survey](https://mlsanity.com/paper/2309.01038/) (2023)
- [Truncated Matrix Power Iteration for Differentiable DAG Learning](https://mlsanity.com/paper/2208.14571/) (2022)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [On Causally Disentangled Representations](https://mlsanity.com/paper/2112.05746/) (2021)
- [MissDAG: Causal Discovery in the Presence of Missing Data with Continuous Additive Noise Models](https://mlsanity.com/paper/2205.13869/) (2022)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [Reinforcement learning on graphs: A survey](https://mlsanity.com/paper/2204.06127/) (2022)
- [Causal Deep Learning](https://mlsanity.com/paper/2303.02186/) (2023)
- [Automated Scientific Discovery: From Equation Discovery to Autonomous Discovery Systems](https://mlsanity.com/paper/2305.02251/) (2023)
- [FedDAG: Federated DAG Structure Learning](https://mlsanity.com/paper/2112.03555/) (2021)
- [CausPref: Causal Preference Learning for Out-of-Distribution Recommendation](https://mlsanity.com/paper/2202.03984/) (2022)
- [Optimizing NOTEARS Objectives via Topological Swaps](https://mlsanity.com/paper/2305.17277/) (2023)
- [Causal Multi-Agent Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2111.06721/) (2021)

## Abstract (arXiv)

Discovering causal structure among a set of variables is a fundamental problem in many empirical sciences. Traditional score-based casual discovery methods rely on various local heuristics to search for a Directed Acyclic Graph (DAG) according to a predefined score function. While these methods, e.g., greedy equivalence search, may have attractive results with infinite samples and certain model assumptions, they are usually less satisfactory in practice due to finite data and possible violation of assumptions. Motivated by recent advances in neural combinatorial optimization, we propose to use Reinforcement Learning (RL) to search for the DAG with the best scoring. Our encoder-decoder model takes observable data as input and generates graph adjacency matrices that are used to compute rewards. The reward incorporates both the predefined score function and two penalty terms for enforcing acyclicity. In contrast with typical RL applications where the goal is to learn a policy, we use RL as a search strategy and our final output would be the graph, among all graphs generated during training, that achieves the best reward. We conduct experiments on both synthetic and real datasets, and show that the proposed approach not only has an improved search ability but also allows a flexible score function under the acyclicity constraint.